How do I find the rule my agents fail most?

Track tells you which rule your agents fail most, and the answer can be a rule nobody was worried about. You ask in plain words and your agent reads the figures through the reporting tools. See reporting for what each tool does.

1. Make sure your agents send the right fields

Reports can only group by what the agent reports. Ask every agent to send agent_model, tags, and output_ref on each validatecall. Calls without them still count, but they appear as “model not reported” or “no reference sent.”

2. Ask for the failure rate by rule

Which of our Vindex rules fail most often this month? Show the failure rate for
each rule, and for the top three, break it down by rule version and by agent.
Give me the definition of the metric and the date the data runs to.

Your agent calls get_reporting_schema and query_report with the rule_fail_ratemetric. The count of a rule’s judgements includes the times it ran alone and the times it ran inside any rule set. A rule with too few judgements to rate is listed with its count, never as a percentage.

3. Break it down by version and agent

By rule version, each version is its own row, so you can see whether a rewrite helped. By agent and model, you can see whether the failures come from one agent. Test runs are left out unless you ask for them, so the figures reflect real work.

To check whether a change to an agent’s instructions helped, tag its calls, for example instructions:v2, and compare by tag or compare two date ranges.

4. Decide what to change

• Every agent fails the same rule. The rule is probably ambiguous or asks for something nobody can meet. Revise it, test the new version, and set it to active.
• One agent or model fails it.The rule is fine and that agent’s instructions are not. Look at what that agent was told.
• Nothing passes it. Check the rule against a draft you know is good, as in the testing guide.

Reading the figures

Every figure comes with the metric’s definition and the time the data runs to. Ask your agent to say both when it reports a number. Credit figures count only Vindex’s price per critique, not the tokens your agents spent revising.