Calculations run on this device. Scenario values are not sent anywhere — the engine is fully local.
agents calculator · free
Tool-Choice Confusion Matrix
Count chosen vs intended tools across prompts and plot the confusion.
Which two tools does the model keep swapping?
Engine 1.0.0 · calculation
50 %
Higher = tool docs look more alike → more off-diagonal swaps.
Fixes, in order:
1. Sharpen the two most-confused tool descriptions with boundary cases.
2. Add negative examples (“use notebook for scratchpad notes, code_exec for runnable math”).
3. Re-run the matrix — the diagonal should tighten before any new tool ships.
~200 labeled prompts replayed through the router. Rows = intended tool, columns = chosen tool. Diagonal green = correct.
searchcalculatorcode_execnotebook
search43421
calculator54113
code_exec32432
notebook75929
- Overall correct
- 156 / 200
- Worst confusion
- notebook → code_exec9 cases
- Likely cause
- Overlapping docstringsnotebook reads like code_exec
Method
- Rows are intended tools, columns chosen tools across 200 labeled examples; the diagonal is correct picks.
- Worst off-diagonal cell tells you which two tool descriptions overlap — fix boundaries before adding new tools.