Grade what your agent writes, then grade the grader.
Builds one Claude judge for each rule a script cannot check, then tests every judge against outputs you trust and copies with one planted mistake, so you know its catch rate before it gates anything.
In plain English: “How do I know my eval is actually right?”
Inside: The one-rule judge pattern, a planted-mistake test set builder, the trust report runner, and a sample you can run in a minute.
Install it in one line, or paste it in.
~/.claude/skills/ and runs automatically when it is relevant.Connect your context. Set it to your motion.
a written list of the rules your output has to follow, and 10 or more outputs you have already reviewed and trust. Twenty is better.
the judges are Claude models called through the Anthropic SDK.
the starter scripts install Ragas, pinned to versions that work together.
if you have never written your agent's rules down, run `agent-contract` first. A judge is only as sharp as the rule it is given.
anything a regex can check should be checked by a regex. Judges cost money and can drift. Scripts do neither.
REVIEWER does not move. A judge that disagrees with your contract is telling you the contract is vague. Settling that is a person's call. Tuning the judge until the flag disappears hides it.
| Set this | What it is | Default / Example |
|---|---|---|
| RULES | the contract lines that need judgment | the ones your format checker cannot check |
| CLEAN_SET | outputs that already passed a review | 20 or morefrom real runs |
| PLANTS | mistakes planted per judge, by script | 8 to 12in 2 to 4 different styles |
| HOLDOUT | new mistake styles added after the first run, never used for tuning | 3 to 4 per judge |
| JUDGE_MODEL | the model each judge runs on | start on Claude Haiku 4.5move a judge up only when a rerun shows it is needed |
| REPEATS | how many times every judgment runs | 2 to tune5 for a gate |
| CATCH_BAR | the share of planted mistakes a judge must catch | every one on a zero tolerance rule90% or more otherwise |
| ALARM_BAR | the share of clean outputs a judge may fail | under 5%after a person reads each flag |
| CONSISTENCY_BAR | the share of judgments that match across repeats | 0.95 |
| REVIEWER | who reads the disagreements and settles contract questions | the agent's owner. never the judge itself |
Everything the skill does, in full.
Your eval scored the format. Nobody scored the judgment.
A format checker can count words, find the JSON, and catch a banned phrase. It cannot tell you whether the reason an agent gave is a real reason, whether a "tip for the rep" is quietly a ready-to-send email, or whether two sources that disagree got smoothed into one confident answer. Those are the failures that reach a customer.
This skill builds a Claude judge for each of those rules. One rule per judge, pass or fail, with the judge's reason attached. Then it does the part most people skip: it tests the judges. It runs each one against outputs you already trust and against copies with exactly one mistake planted, so you get a catch rate, a false alarm rate, and a consistency score for every judge before any of its verdicts count toward a gate.
A judge is a prompt. It can be wrong in the same ways the thing it grades can be wrong, so it gets an eval of its own.
Built while gating a prospecting research agent in October 2026. The agent writes a one-page brief for a sales rep, and its contract has a dozen rules. A format checker covered the shape. Four rules needed meaning: the reason to reach out names a real cause and a real problem, nothing in the brief is outreach copy, a title that sources disagree on is never marked verified, and the brief never leads with personal trivia.
Planted mistakes made the gap visible: the format checker passed briefs a person would fail. It took several rounds of tuning before the judges could be trusted, and the rounds taught the method:
- Narrow what the judge sees. Shown the whole brief, the reason judge found a cause somewhere else in the brief and passed reasons that had none. Showing it only that one line fixed it.
- Pick the model by measurement. The smallest model was as good as the mid-size one on three rules. On the fourth it kept reading a label as an instruction and raised false alarms on good briefs. That one judge moved up a model. The other three stayed cheap.
- A flag on a "good" output is a question, not an error. Some of the conflict judge's flags were real misses the earlier review had passed. The contract had never said exactly what counts as a disagreement, and the judge found the hole.
What is still unsolved: a judge sees only the output, so it cannot catch what the agent failed to find. Invented facts and prompt injection need the source pages passed in next to the output. That part is not in this kit yet.
A fictional sample ships with this skill: four account research briefs and two judges written for them.
- Copy the
scripts/andexample/folders from this skill somewhere you can run Python 3.10 or newer. - Install and set your key:
python3 -m venv .venv && .venv/bin/pip install -r scripts/requirements.txt export ANTHROPIC_API_KEY=... - Build the test set and run the judges:
.venv/bin/python scripts/build_set.py --judges example/judges.py --clean example/clean --out judge-run .venv/bin/python scripts/run_judges.py --judges example/judges.py --out judge-run - Open
judge-run/results/<time>/report.md. Read the table, then read every disagreement line under it. That list is where the learning is.
The sample is about 40 judge calls and costs a few cents.
Your first exercise is already in the report. On the run before this was published, the because judge passed both held out mistakes, a Because that only says the company "sits outside our size range". The rubric says that should fail, and the judge decided it meant "no problem maps". Decide which one is right, change the rubric or the contract to match, and rerun. That loop is the whole skill in miniature.
- 1Step 1: Split the rules
Go through the contract line by line and sort every rule into one of two piles.
- Script: anything checkable by shape. Word counts, required fields, valid JSON, a value in a closed set, a banned string. These go in a format checker, and the checker runs first. A failure there stops the case before a judge is ever paid for.
- Judge: anything that needs meaning. Is the cause real. Is this a sentence to the customer or advice to the rep. Do these two sources disagree.
If a rule lands in neither pile, it is not a rule yet. Send it back to the contract.
- 2Step 2: Write one judge per rule
One rule, one judge, pass or fail. Never one judge asking "is this good, 1 to 10". A narrow question gets the same answer twice, and when a case fails you know which line broke.
Each rubric has four parts:
- The rule, in the contract's own words, with the line number.
- What passes.
- What fails.
- The near misses that must pass. This is where false alarms come from. A proof point the rep will repeat is not outreach copy. A person holding two compatible titles is not a conflict. Write the near misses down or the judge will flag them.
When the rule has steps, ask for the steps: read the verification word, look for any other title, fail if both. A judge walking named steps is easier to check than one giving a verdict from a single read.
- 3Step 3: Show the judge only what it should judge
Cut the input before you write a cleverer prompt. A judge that sees the whole output will borrow evidence from lines the rule is not about. Give it a
view()that passes only the part the rule covers.The exception runs the other way: a judge cannot see what is not there. If a rule depends on a source page or the original request, pass that in too, or say plainly that the judge does not cover the rule.
- 4Step 4: Build the test set
Three parts, built by
build_set.py:- Clean outputs that already passed a review. A judge should pass them.
- Planted mistakes: copies of clean outputs with exactly one rule broken by a scripted edit. The right answer is known the moment you write the edit, so you can measure a catch rate without anyone labelling failures by hand. Plant in a few different styles, and include some a regex would miss.
- Held out mistakes: new kinds of breakage you add after the first run and never use to tune a rubric. A judge that catches these learned the rule. One that catches only the originals learned the test.
- 5Step 5: Run it and read every disagreement
run_judges.pyruns every judge on every case, REPEATS times, and writes a report: false alarms on clean outputs, catches on planted and held out mistakes, what your format checker caught of the same mistakes, and consistency.Then a person reads every disagreement, with the judge's reason next to it. There are three kinds:
- The judge is wrong. Tighten the rubric, add the near miss, or narrow the view, then rerun.
- The earlier review was wrong. The "clean" output really breaks the rule. Count it as a catch and fix the agent.
- The contract is vague. Both readings are defensible. The REVIEWER rules, the rule is written into the contract and the agent's prompt, and the rubric follows the contract.
- 6Step 6: Choose each judge's model
Start every judge on Claude Haiku 4.5. Move one judge to a larger model only when a rerun shows the small one failing that rule, and record why next to the judge. Most rules do not need the larger model. Some do, and you only find out by measuring.
- 7Step 7: Gate on it, and keep it honest
A judge counts toward a gate once it clears CATCH_BAR, ALARM_BAR, and CONSISTENCY_BAR. After that:
- Every real production failure becomes a new planted case. The test set grows; it never restarts.
- A rubric change means a rerun. A verdict from an older rubric is stale.
- Keep the report next to the verdict. A score means nothing without the judges that produced it and the test that earned them trust.
- Every judge maps to exactly one rule, named by its contract line.
- Nothing a script can check is sent to a judge.
- Every rubric lists the near misses that must pass.
- Every judge was run against clean, planted, and held out cases before its verdicts counted.
- Held out cases were added after the first run and never used to tune a rubric.
- The report lists every disagreement with the judge's reason, and a person read them.
- A judge's model was chosen by a measured rerun, not by assumption.
- No judge was tuned to silence a flag that was really a contract question.
- The skill says which rules the judges cannot see, rather than implying full coverage.
- No em dashes anywhere.
JUDGE TRUST REPORT the fictional sample, 2 repeats, 40 judge calls, a few cents
judge false alarms planted held out format checker consistency
on clean caught caught caught the same
because_names_cause_and_problem 0 of 4 4 of 4 0 of 2 2 of 6 1.00
no_outreach_copy 0 of 4 4 of 4 2 of 2 0 of 6 1.00
MODELS
because_names_cause_and_problem on Claude Haiku 4.5
no_outreach_copy on Claude Sonnet 5.5
DISAGREEMENTS, for a person to read
because_names_cause_and_problem, held out, cedar-roofing: missed on both repeats.
the line: "Low fit: Cedar Roofing Co sits outside our size range."
judge: pass. it read the size remark as "no problem maps".
the rubric says restating the size is a fail.
because_names_cause_and_problem, held out, dune-pest: missed, same reason.
OPEN
does "outside our size range" count as "no problem maps"? the judge and the
rubric disagree, so a person decides.
NEXT
if yes, write it into the contract as an allowed form. if no, name it in the
rubric as a fail with this exact example. then rerun the held out cases.
Where an operator takes this next.
A prompt change that passes the format checker and drops a judge's pass rate is the regression you wanted to catch.
Each real miss becomes a script edit that reproduces it. The set grows with the agent.
A writing skill's voice rules are a contract too. Script the mechanical ones, such as dashes and banned words, and judge the rest.
For invented facts, give the judge the source pages the agent read alongside its output, and ask whether each claim is supported by them.
One skill is the on-ramp.
A single skill does one job. Chained into a playbook, or run as a full build, it becomes a system. Here is where this one plugs in.