Reporting

Track opens the stats on how your agents do against your rules. You ask your agent in plain words, and it reads fixed metric definitions through three tools. Nothing runs against raw data. See pricing for what Track costs.

The tools

Tool	What it does
`get_reporting_schema`	Returns the metrics with their definitions and versions, the dimensions and filters you may use, the limits on a request, the earliest date with data, and notes on recent changes. An agent can skip this call while it holds the current schema version.
`query_report`	Returns rows for one metric. It takes the metric, up to two dimensions, filters, from and to dates, a period (day, week, or month), compare (none or prior), sort, limit, and a cursor for the next page.
`get_benchmarks`	Compares your figure on a public rule or rule set with anonymised peers. It takes the metric, one public rule or rule set, a month, and optionally by_agent_model.

Every answer covers your company only. Without dates, a request uses a recent window, and the schema states the longest range allowed.

Metrics

The schema is the full list, with definitions for your role. These are the ones to know first.

Metric	What it tells you
`first_attempt_pass_rate`	How often an agent's first try passes.
`pass_rate`	How often a critique passes.
`attempts_to_pass`	How many attempts a session takes to pass.
`rule_fail_rate`	How often a rule fails, whether it was called alone or inside a rule set. By rule version, each version is its own row, so you can see whether a new version helped.
`cap_rate`	How often a session ends at the attempt cap.
`credits_spent`	What your critiques cost, in credits, with the dollar value. It counts only Vindex's per-critique price, not the tokens your agent spent revising.
`credits_per_passed_session`	Whether the spend bought passes.
`validation_log`	One row per critique: its validation ID, time, member or agent, your output_ref, the rule versions that ran, the result, and the failed rules. It never includes the output text.

Dimensions you can break a metric down by include rule, rule version, rule set, agent, agent model, tag, member, and time period. The agent model is what each agent reported when it called validate. Vindex does not check it, and the schema says so.

How figures are reported

Every figure comes with the metric’s definition and the time the data runs to (data_as_of). Agents are told to give both when they report a number.
A rate over too few sessions or calls is returned as “too few to rate”, with its count, never as a percentage.
A period with no critiques returns an empty state that says so, never a zero rate.
With compare set to prior, each rate also returns the equal-length window before it and a trend: improving, declining, or stable.
Test runs are left out unless you include the test status.
Comparisons across two rule or rule set versions are returned by version, not merged, so a rule change is not mistaken for an agent change. If a comparison spans a change in the engine, the answer says so.
Long answers come back in pages with a cursor, and a cursor expires.

Questions it answers

Which rules fail most often, and did version 4 of the tone rule help?
Which agent and model pass first time most often on the same rule set?
Did the new agent instructions help? Tag calls with instructions:v2 and compare by tag.
Where did the credits go this week, and did the spend end in passes?
Which customer emails failed the privacy rule set in September? Match the rows to your own records by output_ref.

To make these answerable, ask your agents to send agent_model, tags, and output_ref on every validatecall. Calls without them are reported as “model not reported” or “no reference sent”.

Who sees what

Company admins see every member’s figures. A Company user asking for the member breakdown gets their own row, and the answer says other members’ figures are for Company admins. A filter on something your company cannot see returns an empty result, never “forbidden”. A company that is not on Track is refused and told what Track shows.

Benchmarks

get_benchmarkscompares your figure on a public rule or rule set with a group of other companies that use it. You get your figure, the peer median, and your quartile. It never gives another company’s figure, a list of peers, or how large the group is. A comparison comes back only when enough other companies use the rule or rule set to keep every company anonymous. Otherwise you get NOT_ENOUGH_PEERS, with nothing about the group. A company-only rule or rule set returns BENCHMARK_NOT_AVAILABLE.

Refusals

Each refusal names what was wrong and what is allowed, drawn from the schema, so an agent can retry without asking. The codes include NO_CREDENTIAL, MEMBER_INACTIVE, UNKNOWN_METRIC, DIMENSION_NOT_ALLOWED, TOO_MANY_DIMENSIONS, RANGE_TOO_LONG, and RATE_LIMITED. Reporting tools charge no credits per query.

See also critique results, which records the data these reports read, and the MCP tools.