What Is Observability?
Observability is Bluejay’s advanced monitoring capability, designed to provide deep visibility into your voice agents’ interactions with actual customers. Unlike simulations, observability tracks live, real-time production calls, allowing immediate and continuous performance analysis.Key Benefits
Bluejay’s observability feature delivers critical advantages, enabling you to:-
Monitor Real Customer Interactions
- Capture and analyze actual agent-customer calls, providing authentic insights that simulations alone cannot replicate.
-
Evaluate Performance in Real-Time
- Instantly measure your agent’s effectiveness against genuine customer scenarios, ensuring swift and informed adjustments.
-
Detect Anomalistic Calls
- Automatically identify unusual or problematic interactions, including calls with extended latency, excessive agent errors, miscommunications, or other anomalies that previously would have gone unnoticed.
Features
Bluejay’s observability capabilities include robust tools to enhance your understanding and control of live call interactions:-
Real-Time Performance Dashboards
- View immediate, actionable metrics and KPIs from live calls, including agent accuracy, latency, conversational flow, and adherence to defined objectives.
-
Anomaly Detection and Alerts
- Leverage sophisticated analytics to instantly surface anomalous calls, ensuring rapid issue detection and timely intervention.
-
Detailed Call Analysis
- Access individual call logs, transcripts, and recordings from live interactions, enabling comprehensive qualitative reviews.
-
Trend Analysis and Reporting
- Observe performance trends and generate detailed reports, highlighting improvements, regressions, or emerging issues over time.
Getting Started
To integrate observability into your voice agents and start evaluating calls with the evaluate endpoint, see the Observability cookbook. It walks you through submitting call data (audio or transcripts), retrieving evaluation results, and using custom metrics and webhooks.Included Metrics
Observability evaluations return the following metrics:| Metric | Description |
|---|---|
| Custom metrics | Domain-specific performance indicators you define via the API or Bluejay dashboard. Use them to measure what matters for your use case on every call. |
| Hallucination | Indicates when the agent gives incorrect or fabricated information relative to the knowledge base. Includes reasoning when detected so you can pinpoint and fix issues. |
| Redundancy | Flags when the agent says the same or superfluous information. Includes reasoning so you can trim repetition and tighten responses. |
| Pronunciation | Score for how clear and understandable the agent’s pronunciation is. Includes reasoning when issues are found so you can improve clarity. |
| Predicted Word Error Rate | Estimates transcription accuracy for the call from the provided transcript and audio. Use it to gauge recording and ASR quality. |
| Predicted Gender | Predicts speaker gender. |
| Predicted Age | Predicts speaker age. |
| Predicted Language | Predicts the conversation language (e.g. English). |
| Detailed latency | Breaks down latency into average agent latency, average customer latency, average punctuation latency, and time to first agent utterance. Use to find and fix delays. |
| Turn Count | Counts total speaking turns in the call. Use to compare conversation length and structure. |
| Agent Turn Duration | Measures average length of each agent turn in milliseconds. Use to tune response length and pacing. |
| Avg Agent Words/Turn | Averages words per agent turn. Use to spot overly long or short responses. |
| Avg Agent WPM | Measures agent words per minute (speaking rate). Use to align pace with your brand and accessibility goals. |
| Agent Speaking % | Shows the share of total talk time the agent spoke. Use to balance turn-taking and avoid overtalking. |
| Interruption Counts | Tracks how often the agent or customer interrupted. Use to improve turn-taking and reduce overlap. |
| Avg Customer Words/Turn | Averages words per customer turn. Use to understand customer verbosity and adjust agent behavior. |
Why Observability Matters
Observability transforms your operational strategy by providing critical, real-time insights that simulations alone cannot fully capture. It helps you:- Quickly identify and resolve customer interaction issues before they escalate.
- Gain unprecedented visibility into customer-agent dynamics, improving customer satisfaction and loyalty.
- Continuously refine and optimize your agents based on live data, ensuring consistent, high-quality performance.