Agent Reflection · Comparison
Your evals ran last night. The bad output went out this morning.
Evaluation platforms test an agent against a fixed set of examples before a change ships. Agent Reflection checks each real output while the agent works. Here is what each does, where each stops, and why a team that owns standards usually wants both.
What evaluation platforms do well.
Evaluation platforms are built for one question: did this change make the agent better or worse? They run a prompt, a model, or a whole agent against a set of examples, score the results, and show what moved. That is the right way to compare two versions before release, to catch a regression in CI, and to keep a record of how an agent changed over time.
Many let you write your own scorers, from simple checks to model-based judges, and some score live traffic as well. If your problem is choosing between models or catching regressions, use one. Agent Reflection does not replace it.
Where a test set stops.
A test set holds only the cases someone thought to include. A reply that breaks your promises policy in a way nobody wrote down passes every run. And scores usually land in a dashboard after the fact, so the agent that wrote the bad output often does not learn why it was wrong.
The standard itself often lives inside the evaluation setup, written by engineers in a form the people who own the standard cannot read. When the brand guide changes, someone has to find and edit the scorer.
What Agent Reflection does instead.
Agent Reflection keeps the standard in a catalog of plain-language rules, owned by whoever owns the standard. The agent reads the rules before it writes, calls a critique on the draft, and gets back pass or fail for each rule with the passage that broke it. It revises in the same task and runs the critique again.
Side by side.
| What you need | Evaluation platforms | Agent Reflection |
|---|---|---|
| Run a fixed test set before a release | Yes, the core job | Not what it is for. A critique checks one output at a time |
| Compare two prompts or models on the same data | Yes | Tag each critique with an agent version and compare pass rates by tag, with Track |
| Check an output while the agent is still working | Some tools can score live traffic | Yes. The agent calls the critique and revises in the same task |
| Give the agent the passage that broke the standard | Depends on the scorer | Yes. The failing passage is quoted |
| Where the standard lives | In the evaluation setup, usually edited by engineers | In a catalog of Markdown rules, edited by whoever owns the standard |
| The agent reads the standard before it writes | Not usually | Yes. Agents list and read rules for free |
| Reporting | Scores across test runs | Pass rates, rule fail rates, and credits per passed session, with Track |
| Cost | Varies by tool | $0.03 per critique, and a new account starts with 100 free critique credits |
Evaluation platforms differ. The middle column describes the core job of running a test set. This page describes a category in general, not any one product.
Use both.
Run an evaluation suite to decide whether a new prompt or model is ready. Run rules on the real outputs your agents send, to check them against the standards your company owns. Tag each critique with the agent version, and Track compares versions on live work. A rule that keeps failing is a good candidate for a new test case.
Questions.
- Is Agent Reflection an evaluation platform?
- No. An evaluation platform runs a test set to compare versions. Agent Reflection checks one real output against your rules while the agent works, and returns pass or fail for each rule with the failing passage quoted.
- Does the agent see the result while it works?
- Yes. The result goes back to the agent, which revises the quoted passages and runs the critique again, up to the attempt cap.
- What does a critique cost?
- $0.03 per critique, and a new account starts with 100 free critique credits. Listing and reading rules is free.