KPI Explorer
The KPI Explorer provides detailed time-series analysis of AI services performance with customizable filtering. You can track usage patterns, spot latency issues, monitor guardrail activity, and compare performance across models, agents, users, and services — all from a single view.
AI Observability metrics are automatically collected from Amorphic operations once every hour. These metrics are available only in AI-enabled environments and cover data from the time Trace is deployed.
Getting Started
- Navigate to the AI Observability section
- Go to the KPI Explorer tab
- Check data sync info
- Pick a KPI category tab depending on what you want to analyze
- Set granularity and time range
- Filter producers and metrics
- Open expanded view of a KPI
Data Synchronization
The last sync timestamp at the top of the page shows when metrics were last refreshed. Data updates every hour automatically. If the timestamp is more than an hour old, click Refresh to pull in the latest metrics.
If a sync fails, notification emails are sent to users who have email subscriptions enabled in Amorphic. For help, reach out to the Amorphic support team.
Available KPI Categories
Each KPI card shows an info icon — hover over it for a quick summary of what the KPI measures and how to interpret it.
Token Metrics
Track token consumption across AI services. These KPIs help you understand usage patterns and compare model efficiency.
| KPI | Producers | What it shows | Available Metrics |
|---|---|---|---|
| Tokens by User by Model | User, Model | Shows token usage broken down by which user made a request and which model handled it. Helps in spotting heavy users or identify which models consume the most tokens. | Sum, Average |
| Tokens by Agent | Agent | Shows total tokens consumed by each AI agent over time. Useful for comparing workloads across agents and identifying which ones are the most resource-intensive. | Sum, Average |
| Tokens by Service | Service | Shows token consumption across different Amorphic AI services. Helps you understand which services are driving the most usage. | Sum, Average |
| Input/Output Token Ratio by User | User | Shows the ratio of input tokens (prompts) to output tokens (responses) per user. A high ratio may point to very verbose model responses for a particular user. | Average |
| Input/Output Token Ratio by Agent | Agent | Shows the ratio of input tokens to output tokens for each agent. Helps you understand how efficiently each agent generates responses relative to the prompt size it receives. | Average |
Performance Metrics
Monitor how fast AI services respond. Use these KPIs to catch slowdowns, compare model performance, and identify bottlenecks in workflows.
| KPI | Producers | What it shows | Available Metrics |
|---|---|---|---|
| Latency by Model | Model | Shows end-to-end response time for each model. Helps you compare which models are consistently faster and identify any that are underperforming. | Average, Min, Max |
| Latency by Agent | Agent | Shows end-to-end response latency per agent. Useful for finding agents with high or unpredictable latency that may be affecting user experience. | Average, Min, Max |
| Latency by Knowledge Base | Knowledge Base | Shows response time for KB-backed queries, broken down into chunk retrieval time, model inference time, and total latency. Hover over a bar to see the split for each knowledge base. | Average, Min, Max |
| Time to First Token by Model | Model | Shows how long each model takes to return the very first token in a streamed response. Lower values mean users see a response start appearing sooner, which improves perceived speed. | Average, Min, Max |
| Time to First Token by Agent | Agent | Shows how long each agent takes to return the first token in a streamed response. Useful for comparing agents that may use different models or streaming configurations. | Average, Min, Max |
| Requests by User by Model | User, Model | Shows the total number of requests broken down by user and model. Helps you track who is making requests, how frequently, and which models they are using. | Sum |
| Requests by Service | Service | Shows total request volume for each Amorphic AI service. Useful for understanding how load is distributed across services. | Sum |
| Catalog Search Latency | N/A | Shows how long catalog semantic search operations take. Helps you track whether users are getting search results quickly and catch any degradation over time. | Average, Min, Max |
Safety & Operations
Monitor guardrail activity, sync job health, and catalog indexing operations. These KPIs help ensure your AI environment is running reliably and safely.
| KPI | Producers | What it shows | Available Metrics |
|---|---|---|---|
| Guardrail Hit Rate | Guardrail | Shows how often each guardrail engine is intervening on requests. The sum aggregation gives you the total number of requests blocked, while the average gives you the percentage of requests blocked out of all requests. | Sum, Average |
| Sync Duration by Knowledge Base | Knowledge Base | Shows how long sync jobs take for each knowledge base. Spikes in duration may indicate large document batches or connectivity issues. | Average |
| Sync Duration by Service | Sync Service | Shows average sync job duration across sync-enabled Amorphic services. Helps you compare sync efficiency and spot services that are consistently taking longer than expected. | Average |
| Sync Documents by Knowledge Base | Knowledge Base | Shows the number of documents scanned, indexed, and failed to sync during each KB sync operation. Hover over a bar to see the breakdown per knowledge base. | Sum |
| Catalog Indexing Duration | N/A | Shows how long each catalog indexing run takes. Helps you spot unusually long indexing jobs that may need attention. | Sum, Average |
| Catalog Indexing Asset Details | N/A | Shows how many assets were processed during catalog indexing. Hover over a bar to see the count of assets that succeeded, failed, or were scanned in total for that run. | Sum |
Time Range and Granularity
Granularity controls how your data is grouped and displayed over time. By default, the KPI Explorer opens with Daily granularity showing the last 7 days, including today.
-
Hourly granularity is useful for short activity windows, with a maximum selectable range of 7 days within the past 2 months.
-
Daily granularity is suited for general trend analysis, with a maximum selectable range of 2 months within the past 6 months.
-
Monthly granularity is suited for long-term comparisons, with a maximum selectable range of 12 months within the past 12 months.
Filtering by Producers
Producers are the dimensions used to break down and analyze KPI data, such as User, Model, Agent, or Service. The available producer filters depend on the selected KPI.
Single-producer KPIs (e.g., Tokens by Agent): You can select up to 5 producers to compare.
Two-producer KPIs (e.g., Tokens by User by Model): One dimension can include up to 5 selections, while the other supports 1 selection to keep comparisons focused and readable.
Example: Compare up to 5 users for a specific model, or compare up to 5 models for a specific user.
If a Producer filter is available, All is selected by default, which shows an aggregated trend across all producers as a single line.
Metric Selection
Depending on the KPI, you can switch between different aggregation types to change how data points are calculated for each interval:
- Sum shows the total volume over the selected period, which is most useful for usage and count metrics.
- Average shows the mean value per interval and works well for performance metrics like latency.
- Minimum/Maximum show observed extremes, helping you identify best-case performance and catch occasional spikes.
Some KPIs use a predefined aggregation and therefore do not display aggregation toggles. Hover over the info icon for metric details.
Expanded View
Click the Expand icon on any KPI card to open a larger view of that metric. The expanded view is useful when you want to inspect data points more closely.
