1 · Three verdicts
- PASS — every check on the resulting state held, in every attempt.
- FAIL — a check failed: wrong amount, wrong customer, access removed too early, a forbidden action, a double effect.
- INCONCLUSIVE — the state could not be trusted (agent timeout, unreadable state). Never counted as a success.
The verdict comes from the simulated Stripe, not from the reply. An AI note may comment on tone or clarity, but it never turns a FAIL into a PASS.
2 · Expected vs Observed
Open a failed scenario. The Expected vs Observed table lists each field COLVO checked — for example a refund amount or cancel_at_period_end — what it should be, what the simulated Stripe actually shows, and a ✓ or ✗.
3 · The conversation and the proposals
Below the table is what the agent said. If the reply claims an outcome the state does not show, COLVO flags it: “Reply claims an outcome — state disagrees.” The proposals the agent made, with their decisions, show whether it asked for the wrong thing or asked twice.
4 · Attempts
Each scenario runs several attempts; it passes only if every attempt passes. “2 / 3 attempts passed” means the agent is inconsistent — usually a prompt that leaves room for guessing.
5 · Fix and compare
Change the prompt or the tool, save a new build and run again. On the Reports page, compare the two runs to see which scenarios you fixed, which broke and which are still failing.