AI agents analytics
EARLY ACCESSAI agents analytics enables you to monitor performance and usage metrics for your AI agents and tools. Track agent effectiveness, response times, interaction volumes, and tool utilization to optimize your AI-powered customer service operations.
AI agents analytics consists of two main categories:
- AI agent performance - Monitor AI agent performance, tool usage, and session activity. Access specialized metrics through three dedicated tabs: AI agents, Tools, and Sessions.
- Knowledge agent analytics - Track knowledge base performance and content effectiveness. Analyze how AI agents utilize your knowledge resources to provide accurate responses.
AI agent performance
AI agent performance provides comprehensive insights into your AI agents, their tool usage, and session activity. Monitor performance metrics, identify optimization opportunities, and track how effectively your AI agents handle customer interactions.
At the top of the page, you can switch between the following tabs:
- AI agents
- Tools
- Sessions
AI agents
The AI agents tab displays performance metrics for all your AI agents. Monitor agent effectiveness, response times, and interaction volumes to ensure optimal performance.
Summary
| Metric | Description |
|---|---|
| Total agents | Total number of AI agents in your system. |
| Average response time | Average time taken by AI agents to respond to customer interactions (in milliseconds). |
| Total interactions | Total number of interactions handled by all AI agents. |
| Total failures | Total number of failed AI agent interactions. |
Performance by agent
The Performance by agent section provides detailed metrics for each AI agent:
| Column | Description |
|---|---|
| Agent name | Name of the AI agent. |
| Interactions | Number of interactions handled by this agent. |
| Average response time (ms) | Average response time for this agent in milliseconds. |
| Failures | Number of failed interactions for this agent. |
| Success rate | Percentage of successful interactions for this agent. |
Tools
The Tools tab displays performance metrics for all tools used by your AI agents. Monitor tool effectiveness, call volumes, and success rates to ensure your AI agents have access to reliable and efficient tools.
Summary
| Metric | Description |
|---|---|
| Total tools | Total number of tools available to AI agents. |
| Total tool calls | Total number of times tools have been called by AI agents. |
| Average call time | Average time taken to execute tool calls (in milliseconds). |
| Overall success rate | Percentage of successful tool calls across all tools. |
Performance by tool
The Performance by tool section provides detailed metrics for each tool:
| Column | Description |
|---|---|
| Tool name | Name of the tool. |
| Total calls | Number of times this tool has been called. |
| Average call time (ms) | Average execution time for this tool in milliseconds. |
| Failed calls | Number of failed calls for this tool. |
| Success rate | Percentage of successful calls for this tool. |
Sessions
The Sessions tab displays a table of all AI agent sessions. Use this tab to review individual session details and track interactions.
| Column | Description |
|---|---|
| Session ID | Unique identifier for the session. |
| AI agent | The AI agent that handled the session. |
| End user | The end user who participated in the session. |
| Date | Date of the session. |
Knowledge agent analytics
Knowledge agent analytics enables you to monitor performance of your knowledge agents and identify opportunities for improvement. Track how effectively your AI agents utilize knowledge resources, analyze response patterns, and optimize documentation based on real usage insights.
There are two task types:
- Analyze user traffic - Analyze user messages and knowledge agent responses.
- Evaluate - Evaluate knowledge agent performance.
Analyze user traffic
Analyze messages your users sent in the selected date range and get insights into knowledge agent performance.
The report provides a detailed list of user messages, the agent response for each message, and the context on which the answer is based so you can better understand why it answered a certain way. It also generates topics.
You can analyze up to 4,000 messages. If more messages exist in the selected period, a random sample is taken across the entire period.
Create task
To create a task:
- Select a knowledge agent from the drop-down menu.
- Enter task name.
- Select either chatbots or senders from the drop-down menu to filter the analysis. You can only select either chatbots or senders, not both.
- Choose the From and To date.
- Select Run task.
After you run the task, it appears with the details in the task table below.
Evaluate
Evaluate tests your knowledge agent with premade questions. It provides a list of all agent responses for those questions and the context on which each answer is based, so you can understand why the agent answered a certain way.
To run an evaluation, upload a .csv file that contains only one column with test questions for the knowledge agent. These questions should be questions your users are likely to ask when interacting with your business. You can download a CSV example to see the expected format.
Create task
To create a task:
- Select a knowledge agent.
- Enter task name.
- Select Run task.
Tasks
The task table keeps a history of the last 20 tasks per task type.
| Column | Description |
|---|---|
| Task name | Name of the task. |
| Knowledge agent | The knowledge agent that was analyzed. |
| Created at | Date and time when the task was created. |
| Status | Current status of the task (In progress, Failed, or Completed). |
You can view a detailed report or download it for each completed task.
Detailed report
You can access the detailed report of a task by selecting the first icon in the Actions column of the Tasks table. You can also select the download icon to save it to your PC.
This opens a page with two tabs: Overview and Insights.
Overview
The Overview tab displays the following summary metrics:
| Metric | Description |
|---|---|
| Total interactions | Total number of interactions analyzed. |
| Avg response time | Average time the AI assistant took to respond. |
| Success rate | Percentage of successful responses. |
| Documents retrieved | Number of different documents used to generate responses. |
| Avg interactions per session | Average number of interactions in each session. |
| Avg interactions per user | Average number of interactions per user. |
The Overview tab also includes:
- Insight distribution - Breakdown of different insight categories.
- Response latency - Hourly latency trends.
Insights
The Insights tab categorizes interactions into the following types:
| Category | Description |
|---|---|
| Successful answers | Interactions where the AI assistant provided correct and helpful responses. |
| Improve documentation | Interactions where the documentation could be improved to provide better answers. |
| Not documented | Interactions about topics that are not covered in the documentation. |
| Unexpected messages | Interactions with unexpected or off-topic user messages. |
| Unusual examples | Edge cases and unusual interaction patterns. |
Select a category tab to view the individual interactions that fall under that category.