Lesson: API & Automation · Topic 2 of 2
Lesson summary
Running tests manually is fine when you are getting started, but as your agent matures you will want testing and monitoring to happen on autopilot. This topic covers Workflows (automated test pipelines), Schedules (cron-based recurring runs), CI/CD integration, and the voice provider and transport integrations that connect Bluejay to your production stack.Objectives
- Build a Workflow that chains simulation runs, evaluations, and notifications
- Schedule a Workflow to run on a recurring basis
- Understand how to integrate Bluejay into a CI/CD pipeline
- Connect voice provider and simulation transport integrations
Video walkthrough
Video coming soon. Follow the written walkthrough below in the meantime.
Walkthrough
Understand Workflows
A Workflow is a graph of connected steps that Bluejay runs for you. Each step is a node, and you connect them in sequence or with branches. Common node types include:
- Simulation run — Trigger a simulation
- Evaluation — Score a conversation or set of conversations
- Notification — Send a message (Slack, email, webhook)
- Branch — Take different paths based on a condition (e.g., if a metric fails, send an alert; if it passes, continue)
Set up a Schedule
A Schedule attaches a cron expression to a Workflow so it runs automatically at a regular interval.Common schedules:
- Daily at 9 AM — Run your regression suite every morning before the team starts work
- Hourly — Continuously monitor a high-traffic agent
- Weekly on Monday — Run a full test plan at the start of each sprint
- Open the Workflow you want to automate
- Click Add Schedule
- Set the cron expression or pick from the presets
- Save — the Workflow will now run automatically at the times you specified
Integrate with CI/CD
You can trigger Bluejay simulations from your CI/CD pipeline so every code change gets tested automatically. The pattern:
- Your pipeline deploys a new version of your agent
- A script calls the Bluejay API to queue a simulation run
- The script polls for results (or uses a webhook to wait)
- If metrics pass, the pipeline continues; if they fail, it blocks the deploy
Connect voice provider integrations
If your agent runs on a third-party voice provider, you can connect it to Bluejay for automatic observability (production call scoring) and simulation transport.Observability integrations (production monitoring):
Each guide walks you through the connection step by step. Most take under ten minutes to set up.
Simulation transport integrations (how Bluejay calls your agent during tests):
| Transport | When to use it |
|---|---|
| Telephony | Standard phone calls (PSTN) |
| SIP | SIP trunking (recommended for phone-based agents) |
| LiveKit | WebRTC voice via LiveKit |
| WebSockets | Streaming text or audio (CHIRP protocol) |
| HTTP Webhooks | Request/response chat agents |
| Pipecat | Pipecat Cloud agents |
Activity
Knowledge check
What is a Workflow in Bluejay?
What is a Workflow in Bluejay?
A Workflow is an automated pipeline made up of connected steps (nodes). It can chain simulation runs, evaluations, notifications, and branching logic together so multi-step test processes run without manual intervention.
When would you use a Schedule?
When would you use a Schedule?
Use a Schedule when you want a Workflow to run automatically at regular intervals — for example, running your regression suite every morning or monitoring a production agent hourly.
How do you integrate Bluejay into a CI/CD pipeline?
How do you integrate Bluejay into a CI/CD pipeline?
Your pipeline calls the Bluejay API to queue a simulation run after deploying a new agent version. It then waits for results (via polling or webhooks). If the metrics pass, the deploy continues; if they fail, the pipeline blocks. The GitHub Actions cookbook has ready-to-use examples.