Your First Simulation

Create, configure, and run a simulation from scratch using the Bluejay dashboard.

Lesson: Testing with Simulations · Topic 1 of 2

Lesson summary

A simulation is how you test your AI agent before real customers talk to it. You set up a fake conversation — complete with a synthetic caller, a scenario, and grading criteria — and let Bluejay run it for you. This topic walks you through building and running your very first simulation, step by step.

Objectives

• Create a new Simulation attached to an Agent
• Add a Digital Human with a scenario script
• Attach Custom Metrics to define what “success” looks like
• Run the Simulation and review the results

Video walkthrough

Video coming soon. Follow the written walkthrough below in the meantime.

Walkthrough

1. Navigate to Simulations

   Open the Simulations page from the sidebar and click New Simulation. Select the Agent you created in Lesson 1.

   A Simulation is always linked to one Agent. This tells Bluejay which AI agent to call during the test.

2. Configure the simulation

   Give your Simulation a descriptive name — something that tells you what you are testing. For example, “Happy-path order check.”

   Next, choose a connection type that matches how your agent communicates:

   Connection type	When to use it
   Telephony	Your agent answers phone calls
   SIP	Your agent uses SIP trunking
   LiveKit	Your agent uses WebRTC via LiveKit
   WebSocket	Your agent streams text or audio over a WebSocket
   HTTP Webhook	Your agent handles request/response chat

   If you are not sure which one to pick, start with the type that matches your production setup. You can learn more in Simulation Integrations.

3. Add a Digital Human

   Click Add Digital Human and fill in:

   • Name: A friendly label (e.g., “Impatient Customer”)
   • Scenario script: A description of what the caller wants to do. Write it in second person: “You are calling to check the status of order #12345. You are in a hurry and want a quick answer.”
   • Persona traits: Optional details like tone, language, or patience level

   The scenario script is the most important field. It tells the Digital Human exactly what to say and do during the call.

4. Attach Custom Metrics

   Select or create Custom Metrics that define success for this conversation. For example:

   • “Did the agent confirm the order number?” (pass/fail)
   • “Was the caller greeted within the first 10 seconds?” (pass/fail)
   • “Overall conversation quality” (qualitative: poor / fair / good / excellent)

   Metrics are the grading rubric for the conversation. Without them, you can read the transcript but you will not get an automatic score.

5. Run the Simulation

   Click Run. Bluejay will start the conversation between your Agent and the Digital Human. Depending on the connection type, the call may take anywhere from a few seconds to a few minutes.

   When it finishes, click into the Simulation Run to see:

   • The full transcript (what was said, turn by turn)
   • Metric scores (pass/fail, qualitative ratings, etc.)
   • Call metadata (duration, latency, turn counts)

Activity

Hands-on exercise: Create a simulation with one Digital Human whose scenario is: “You are calling to check the status of your order. You want to know when it will arrive.” Attach a pass/fail Custom Metric: “Did the agent confirm the order number?” Run the simulation and review the results page. Check whether the metric passed or failed, and read through the transcript to see if the score makes sense.

Knowledge check

What is a scenario script?

A scenario script is a plain-language description of the Digital Human’s goal, constraints, and behavior during a simulated conversation. It drives what the synthetic caller says and does — think of it as stage directions for an actor.

Can you run a Simulation more than once?

Yes. Simulations are designed to be repeatable. Each time you run one, Bluejay creates a new Simulation Run with its own transcript and scores. This lets you track how your agent’s performance changes over time.

Why do you need Custom Metrics?

Without Custom Metrics, Bluejay captures the transcript but cannot score it automatically. Metrics turn a conversation into measurable data — pass/fail, ratings, and specific checks — so you can compare runs and catch regressions.

Next topic

Continue to Topic 2: Advanced Testing Strategies.