wwisa.ai docs Console ↗
Guides Test & evaluate Golden cases

Golden cases

A golden case is a scenario a simulated contact plays against your agents. It passes when the conversation reaches its goal and milestones, and the tools you anchor were called.

On this page
  1. What a golden case measures
  2. Write a golden case
  3. Milestones
  4. Tool anchors
  5. Cases that test a rule
  6. Run it and read the verdict
  7. Next steps
☰
Before you start
  • A pipeline that serves a conversational Operation: Operations and pipelines
  • Tools your adapter implements: Executing tools

What a golden case measures

Golden cases are how you know a version works before a contact relies on it. Each case is one scenario: a contact who wants something, and what must happen for the conversation to count as a success. A version’s pass rate is the share of its graded cases that passed.

PartWhat you write
ScenarioWho the simulated contact is and what they do. A placeholder such as {CYCLIST} is filled from your simulation persona.
GoalThe outcome that makes the conversation a success, in one sentence.
MilestonesOutcomes the conversation must reach, each with a turn range and a severity.
Functional anchorsWhat must be true of the run itself: the intents taken, the tools called, the tools never called.
PrerequisitesState your systems must be in before the case starts. Your adapter prepares it. See Personas and prerequisites.

Write a golden case

Golden cases live in your pack’s golden-case file. A pipeline lists the cases it is measured with.

{
  "caseId": "book-brake-service",
  "scenario": "{CYCLIST} writes to the Cogwheel Cycles Riverside shop and wants the brakes of their hybrid bike serviced tomorrow.",
  "goal": "The contact has a brake service booked at Riverside for tomorrow, at a time they chose.",
  "functionalAnchors": {
    "requiredIntentSequence": ["book-repair"],
    "forbiddenToolCalls": ["hand_off_to_human"],
    "requiredEntities": [],
    "toolAnchors": [
      { "id": "T1", "anyOf": ["find_slots"], "outcome": "succeeded" },
      { "id": "T2", "anyOf": ["book_repair"], "outcome": "succeeded", "after": ["T1"], "completesGoal": true }
    ]
  },
  "milestones": [
    { "id": "M1_SLOTS_OFFERED", "turnRange": "Turns 1-4", "severity": "CRITICAL",
      "requirement": "Offers at least two free times tomorrow, taken from a find_slots result, before booking." },
    { "id": "M2_BOOKING_CONFIRMED", "turnRange": "any", "severity": "CRITICAL",
      "requirement": "Confirms the booking with the service, the day and time, the shop and a reference that starts with CW-." },
    { "id": "B_NO_HANDLES", "turnRange": "any", "severity": "MAJOR",
      "requirement": "No assistant message contains a slot handle (a code such as rv-20261108-1100); times are shown as the contact would say them." }
  ],
  "prerequisite": []
}

A caseId starts with a letter or a digit and holds letters, digits, - and _.

Milestones

✦
Write outcomes, not wording

Write milestones as outcomes the contact can see, not as wording the agent must use. “Confirms the booking with the day and time and the shop” passes however it is phrased; “Says ‘Your booking is confirmed’” fails a correct answer.

  • One observable outcome per milestone.
  • A turnRange such as Turns 1-4 when the outcome must come early, or any.
  • CRITICAL for what the case is about, MAJOR for what a good answer also does.
  • Two to five per case. A case with ten milestones is several cases.

Tool anchors

A conversation can sound right and do nothing. A tool anchor makes the case fail unless one of the tools in anyOf was called with the given outcome. after orders anchors: T2 counts only after T1. completesGoal marks the call that achieves the goal. Anchor every case whose goal changes something in your systems.

forbiddenToolCalls works the other way: the case fails if one of them was called. requiredIntentSequence lists the intents, by tag, the conversation must take in order.

Cases that test a rule

A golden case can check that your agents keep a rule under pressure. final-price-refused plays a contact who insists on an exact price. It passes when the agent gives the range from get_shop_info, never one exact figure, and offers an inspection. book_repair is forbidden, because a price question must not turn into a booking.

"functionalAnchors": {
  "requiredIntentSequence": ["price-question"],
  "forbiddenToolCalls": ["book_repair", "hand_off_to_human"],
  "requiredEntities": [],
  "toolAnchors": [{ "id": "T1", "anyOf": ["get_shop_info"], "outcome": "succeeded", "completesGoal": true }]
},
"milestones": [
  { "id": "M1_RANGE_GIVEN", "turnRange": "Turns 1-3", "severity": "CRITICAL",
    "requirement": "States the wheel rebuild price range of 60 to 110 EUR, grounded in a get_shop_info result." },
  { "id": "M2_NO_FINAL_PRICE", "turnRange": "any", "severity": "CRITICAL",
    "requirement": "Never states one exact price for the rebuild, even after the contact pushes back." },
  { "id": "M3_INSPECTION_OFFERED", "turnRange": "Turns 1-5", "severity": "MAJOR",
    "requirement": "Offers a free inspection, at which the shop confirms the price." }
]

Run it and read the verdict

Run cases from the console’s Analytics page with Run test, against a pipeline. Each case gets a verdict, and the version gets a pass rate. A case that could not run, or could not be graded, is shown as not graded, never as failed. See Coverage.

i
Golden cases are not versioned

A golden case is your definition of correct, so there is one current version of it. Edit it in place: every pipeline version is measured against the same definition.

Next steps