Simulation run on this device. Scenario values are not sent anywhere — the engine is fully local.
research studio · free
Canary Benchmark Builder
Assemble minimal canary suites with stratified coverage.
Which 20 questions catch this regression earliest?
Engine 1.0.0 · studio
20
- Questions
- 20
- Strata
- 4
- Review time
- ~15 min
| ID | Stratum | Prompt shape | Pass criteria |
|---|---|---|---|
| C01 | Happy path | Happy path: a plain request the policy should always pass | pass = matches expected policy outcome |
| C02 | Happy path | Happy path: a plain request the policy should always pass | pass = matches expected policy outcome |
| C03 | Happy path | Happy path: a plain request the policy should always pass | pass = matches expected policy outcome |
| C04 | Happy path | Happy path: a plain request the policy should always pass | pass = matches expected policy outcome |
| C05 | Happy path | Happy path: a plain request the policy should always pass | pass = matches expected policy outcome |
| C06 | Edge case | Edge case: an unusual but legitimate input | pass = matches expected policy outcome |
| C07 | Edge case | Edge case: an unusual but legitimate input | pass = matches expected policy outcome |
| C08 | Edge case | Edge case: an unusual but legitimate input | pass = matches expected policy outcome |
| C09 | Edge case | Edge case: an unusual but legitimate input | pass = matches expected policy outcome |
| C10 | Edge case | Edge case: an unusual but legitimate input | pass = matches expected policy outcome |
| C11 | Adversarial | Adversarial: an input that tries to bypass the policy | pass = matches expected policy outcome |
| C12 | Adversarial | Adversarial: an input that tries to bypass the policy | pass = matches expected policy outcome |
| C13 | Adversarial | Adversarial: an input that tries to bypass the policy | pass = matches expected policy outcome |
| C14 | Adversarial | Adversarial: an input that tries to bypass the policy | pass = matches expected policy outcome |
| C15 | Adversarial | Adversarial: an input that tries to bypass the policy | pass = matches expected policy outcome |
| C16 | Format check | Format check: an input that must produce a specific shape | pass = matches expected policy outcome |
| C17 | Format check | Format check: an input that must produce a specific shape | pass = matches expected policy outcome |
| C18 | Format check | Format check: an input that must produce a specific shape | pass = matches expected policy outcome |
| C19 | Format check | Format check: an input that must produce a specific shape | pass = matches expected policy outcome |
| C20 | Format check | Format check: an input that must produce a specific shape | pass = matches expected policy outcome |
A canary suite is cheap enough to run on every change; full evals are not. Keep every stratum represented.
Method
- Builds a stratified suite across happy path, edge case, adversarial, and format-check prompts with explicit pass criteria.
- A canary suite is sized to run on every change; full evals are not.