Copilot Analytics
The Copilot Analytics section provides detailed monitoring of how users interact with the Copilot. It allows you to review conversation volumes, measure resolution rates, check user satisfaction, and analyze specific conversations. This helps you verify that the Copilot is functioning as expected and identify areas where improvements may be required.
You can access Copilot Analytics from Analyze → Copilot.
Time Range
By default, the analytics dashboard displays data for the last 7 days. This offers a quick way to view the most recent usage.
The time range can be changed using the selector in the top-right corner of the screen:
- Predefined ranges: 30, 60, or 90 days
- Custom range: select a specific start and end date to focus on a particular period
Selecting a shorter range (for example, 7 or 30 days) is useful when you want to check the effect of recent changes or updates. Longer ranges (such as 60 or 90 days) are better when you want to identify patterns over time or compare activity across multiple months.
Metrics
The metrics panel provides three key values:
Total Conversations
The number of conversations initiated during the selected time range. This counts conversations as units, not the number of individual messages exchanged inside them.
Chatting Users
The total number of distinct users who had at least one conversation with the Copilot in the selected time range. Multiple conversations from the same user are counted once.
Resolved Conversations
The number of conversations that ended with a resolved status. This is shown as both an absolute number and as a percentage of total conversations. Please read this section to understand what counts as a resolution.
Visualization
The Visualization section displays graphs that represent conversation activity and user interactions over time.
- You can choose to display either total conversations or chatting users.
- The Show Values option overlays the numerical values directly onto the graph.
- Data can be grouped by different levels of granularity: daily, weekly, monthly, quarterly, or yearly.
This makes it possible to view both short-term fluctuations and long-term patterns depending on the selected range and grouping.
Filters
By default, all conversations are shown. To focus on specific cases, filters can be applied:
Upvotes / Downvotes
- Upvote: the user indicated the response was useful.
- Downvote: the user indicated the response was not useful.
Filtering by votes makes it easier to identify successful responses or to isolate problem areas.
Status
Every conversation is automatically assigned a status:
- Resolved
- Multi-message conversations are considered resolved if the last few messages show neutral or positive sentiment, or if the user does not downvote, does not request a human agent, and leaves the conversation inactive for at least 24 hours.
- Single-message conversations are considered resolved if there is no downvote and no human support request within 24 hours of the Copilot’s reply.
- Escalated: The user explicitly requested a human agent. Escalations indicate that the Copilot did not provide a sufficient answer.
- Pending Evaluation: The conversation has ended, but the user has not yet given feedback (upvote or downvote).
These filters allow you to focus on problem conversations, unresolved interactions, or cases where no feedback has been given.
Conversation Table
Below the graphs, a detailed conversation table lists each recorded conversation individually. The table includes the following columns:
- Conversation Title: the initial user query that started the conversation
- Date: the date when the conversation occurred
- Messages Count: the total number of messages exchanged in the conversation
- Upvotes and Downvotes: the recorded user feedback on response quality
- Status: resolved, escalated, or pending evaluation
Conversation Details
Clicking on a conversation entry opens two tabs with further information:
Feedback Tab
Displays any votes the user gave (upvote or downvote) along with the user’s name, the exact time of the vote, and the message if they left any.
Conversation Tab
This view shows the full transcript of the conversation, including both your messages and Copilot’s responses, so that the entire interaction can be reviewed in context. It gives you a complete picture of how the exchange unfolded and allows you to assess both the user experience and the Copilot’s performance.
In this tab, you can see the entire conversation flow and also review the feedback left on individual messages. Each message may have its own rating, so one response might be upvoted while the next is downvoted.
You can also evaluate the quality of Copilot’s answers by reviewing the sources that were used to generate them. By tracing responses back to their sources, you can better understand why certain answers were produced, improve the underlying source material, and reduce hallucinations.
Finally, if you find that your sources are correct, coherent, and clear, but the Copilot has still generated a hallucinated response, you can report it directly using the Report hallucination button at the bottom of the response.
Improving AI Copilot Based on Feedback
By reviewing user interactions, you can identify gaps in AI Copilot’s responses and refine its resource pool accordingly. If certain topics frequently receive downvotes or negative feedback, consider updating your internal knowledge base or external sources to improve accuracy.
Additionally, analyzing repeated queries can help you discover common pain points. If AI Copilot struggles with specific types of questions, you may choose to:
- Expand the Knowledge Base: Integrate any missing details to ensure comprehensive and up-to-date content.
- Refine Source Selection: Optimize the choice of sources to enhance the relevance and accuracy of responses to user queries.
- Integrate High-Quality External Sources: Include new external sources that offer reliable and in-depth coverage of the relevant topics.
By regularly reviewing and acting on feedback, you ensure AI Copilot continues to evolve and provide better assistance over time.