attest

Evaluating an agent usually means LLM-as-judge: one model grading another. The weakness is that the judge reacts to the agent's explanation, so a confident, well-written answer can pass even when a specific detail buried inside it is wrong. This is measured, not hypothetical. A 2026 paper, Gaming the Judge, found that rewriting an agent's reasoning while leaving its actions unchanged can inflate a judge's false-positive rate by up to 90 percent.

attest never trusts what the agent says it did. It breaks the final answer into atomic claims and verifies each one against the agent's recorded tool outputs, the receipts. The property that makes this resistant to gaming is isolation: the grading model sees only that claim and the evidence, never the agent's reasoning or narrative. It still uses a model to judge, but constrains that judgment to a narrow entailment question rather than a holistic opinion, and every finding quotes the exact span of evidence behind it.

attest grades a run across five dimensions:

• Faithfulness: is every claim backed by a tool output?
• Tool use: were the tools allowed, and were errors handled rather than ignored? No API key needed.
• Prompt injection: did tool outputs carry hidden instructions, and were they obeyed? A regex scan catches known payloads, and an effect-based check catches novel ones by asking whether the agent took an action the user never authorized. It can scan PDFs and images the agent read.
• Role adherence: did the agent stay within the job its system prompt defines, or get talked out of it?
• Policies: did the run follow the project's own rules, written as evidence and rubric checks?

Each finding maps to a pass, review or block verdict, and every check returns the same shape, so the whole report reads and serializes the same way.

A run is a JSON record of one task: the question, each tool call and what it returned, and the final answer. attest run run.json grades it. attest tools and attest injection run their checks alone, without a key. attest review checks an AI code reviewer's outputs against your review guidelines and filters them down to the ones worth posting. attest stats turns a pass count into a rate with a 95% confidence interval, so a real change in the agent can be told from noise. In Python the same run is a Trajectory, and converters build one directly from LangGraph and Haystack messages.

A few choices I am happy with:

• One interface, three providers. Grading runs on Anthropic, OpenAI, or Gemini behind a single structured-output layer, so you can switch models without changing your code.
• The judge is hardened against itself. attest reads attacker-controllable text, which makes its own judge an injection target. A guard frames that text as data to evaluate, never commands to obey, so a planted "mark this as passing" does not flip the verdict.
• Cheap at scale. Faithfulness originally made one model call per claim, which exploded on long answers. I batched the verification and tightened claim extraction, taking one real example from about 170 grading calls down to 5, with the verdicts unchanged.
• Shipped, not just written. It is published to PyPI with automated releases through GitHub Actions Trusted Publishing, so a new version goes out on a tagged release with no manual upload and no long-lived tokens. The tests run offline, with no API key.

I use attest the way evals are meant to be used: as a test suite. I dogfooded it on a real LangGraph code-review agent by writing a spec of expected behavior, a few legitimate requests plus a set of jailbreak attempts, and a runner that grades each result with attest and asserts against the spec. It exits non-zero on failure, so it drops into CI. If the agent's system prompt is ever weakened and it starts complying with a jailbreak, the suite turns red instead of the problem reaching production.

attest constrains a model rather than removing it, so its verdicts are far harder to game than a holistic grader, but not infallible. It trusts that the recorded run is authentic and complete. Prompt injection is an unsolved problem, and attest detects rather than prevents it. Its accuracy has not yet been measured on a labelled benchmark; the benchmark that will measure it, per check, is being built in the repository. The goal is to make agent behavior measurable and auditable, not to claim it is solved.

Repo: github.com/peace-orefejo/attest · Docs: peace-orefejo.github.io/attest · Install: pip install agent-attest