Find out whether the new prompt is actually better.
Runs the old and new prompt on the same inputs and judges each pair in both orders, counting only wins that hold both ways, with a list of anything the new version broke.
In plain English: “Did my prompt change help or break something?”
Inside: The pairwise judges, a pair builder with planted and identical pairs, the runner, and a sample you can run in a minute.
Install it in one line, or paste it in.
~/.claude/skills/ and runs automatically when it is relevant.Connect your context. Set it to your motion.
saved outputs from both versions on the same inputs, named so the same input has the same file name in both folders. Ten inputs is a start. Thirty is better.
outputs a person already reviewed. They can come from either version. The trust test plants mistakes into copies of them.
the ones a script cannot check. If you have never written them, run `agent-contract` first.
the judges are Claude models called through the Anthropic SDK.
the starter scripts install Ragas, pinned to versions that work together.
REGRESSION_RULE does not bend to the win count. A new version that wins 9 of 10 cases and adds outreach copy to one is not ready on a zero tolerance rule. Wins do not cancel a must-never.
| Set this | What it is | Default / Example |
|---|---|---|
| INPUTS | the inputs both versions run on | the same frozen setnever regenerated between versions |
| OLD / NEW | the saved outputs of each version | one file per inputsame file name in both folders |
| RULES | the rules that need judgment, one judge each | the ones your format checker cannot check |
| RULE_KIND | quality, must-always, or must-never, plus zero tolerance | must-never on anything a customer must never see |
| REVIEWED | outputs a person trusts, for the trust test | 4 to 20from either version |
| PLANTS | mistakes planted per rule, by script | 2 styles per rule |
| HOLDOUT | new mistake styles added after the first run, never used for tuning | 1 style per rule |
| JUDGE_MODEL | the model each judge runs on | start on Claude Haiku 4.5move one up only when a rerun shows it is needed |
| WIN_RULE | what counts as a win | wins with old as A and with new as A. anything else is a tie |
| REGRESSION_RULE | what lands on the regression list | any loss on a must-never or must-always ruleand any lean toward old on a zero tolerance rule |
| REVIEWER | who reads the regression list and decides | the prompt's owner. never the judge |
Everything the skill does, in full.
You changed the prompt. The new outputs look better. You read five of them and they are tighter, clearer, more useful.
Then it ships, and a week later someone finds the one thing the old version never did. The rewrite that fixed the format also started slipping a ready-to-send email into the brief. Nobody compared for that, because everybody was comparing for the thing they set out to fix.
This skill runs the old prompt and the new prompt on the same inputs and puts the outputs side by side in front of a Claude judge, one rule at a time. For each input and each rule, the judge answers A, B, or tie. Then it asks again with the two outputs swapped, because judges tend to prefer whichever one they read first. A winner only counts when it wins both times.
You get two answers. Did the change help, rule by rule. And a regression list: every case where the new version is worse on a rule it must never break or must always keep, even if it wins everywhere else.
A judge is a prompt. Before its verdicts count, it gets tested: pairs where you planted the mistake yourself, and pairs where both sides are the same output. If it cannot pick the clean side every time, or cannot call a tie on two copies of the same thing, it is not ready to referee your prompt.
How this differs from brief-prompt-skill: that skill grades one prompt on its own, against a checklist, and compares its pass rate to a naive one-line baseline. This one puts two real versions head to head on the same inputs, asks in both orders, tests the judge first, and lists regressions. Use brief-prompt-skill to write a prompt and get its pass rate. Use this one when you are about to replace a prompt that already works. It builds on agent-judge, which grades one output against one rule, pass or fail.
Built in October 2026 while rewriting the prompt behind a research agent. The old version wrote long briefs where the decision sat at the bottom. The new version put the decision first. The question was whether the rewrite helped, and whether it quietly broke a rule the old version kept.
It was proven on the real outputs of both prompt versions, run on the same inputs, after a test set of planted mistakes and identical pairs showed the judges could be trusted.
What the build taught:
- Asking in both orders caught a judge that would have lied. On the zero tolerance rule, the small model kept naming the output that broke the rule instead of the better one, but only when the bad output came first. Asked once, that reads as a confident verdict. Asked twice, it shows up as a flip. That one rule moved up a model. The others stayed small.
- Say what the letter means. A judge reasoning about a rule that must never break tends to talk about the output that broke it, then answer with that output's letter. One plain sentence, "your answer names the better output", fixed most of it.
- Show both sides the same cut. The two versions wrote in different formats, so "can a rep act on the top of this" only works if the judge sees the same amount of each: the first few dozen words, nothing else.
- Ask for the differences first. A judge counting unsourced claims across two long outputs missed an invented number until the rubric told it to find where the outputs differ, check each difference, and then count.
- Plant mistakes across every output. The first version of the pair builder stepped through the outputs at a stride that skipped most of them, and planted every mistake of one kind into the same output. The trust test looked fine and proved almost nothing. The builder now reaches every output.
What it cannot do yet: it judges what is on the page. If both versions miss the same fact, the judge calls it a tie, and it cannot tell you a fact was missed at all. It also does no statistics. On a few dozen inputs, a difference of two wins is noise, so read the cases behind the counts.
A fictional sample ships with this skill: five account research briefs written by an old prompt and a new prompt, four briefs a person already reviewed, and three pairwise judges.
- Copy the
scripts/andexample/folders from this skill somewhere you can run Python 3.10 or newer. - Install and set your key:
python3 -m venv .venv && .venv/bin/pip install -r scripts/requirements.txt export ANTHROPIC_API_KEY=... - Build the pairs and judge them:
.venv/bin/python scripts/build_pairs.py --judges example/judges.py --old example/old --new example/new --reviewed example/reviewed --inputs example/inputs.jsonl --out pair-run .venv/bin/python scripts/run_pairs.py --judges example/judges.py --out pair-run - Open
pair-run/results/<time>/report.md. Read the trust table first, then the regression list, then the wins.
The sample is 60 judge calls and costs about 15 cents.
Your first exercise is already in the report. The new prompt wins the rule it was written for, and still has a regression. Find it, decide whether it blocks the change, and write down why. That decision is the whole skill in miniature.
- 1Step 1: Run both versions on the same inputs
Freeze the inputs. Run the old prompt and the new prompt on the same model with the same settings, and save each output under its input's name. If one version stops on an input where the other answers (asks a question, refuses, returns nothing), that is a finding a script can see. List it, do not judge it.
- 2Step 2: Split the rules
Anything checkable by shape (word counts, required lines, valid links, a banned string) goes to a script and runs on both sides first. Only rules that need meaning get a judge. Label each one:
- quality: better is better, a loss is worth reading.
- must-always: a loss goes on the regression list. Losing a rule the old version kept is a regression, even when the new version wins elsewhere.
- must-never: a loss goes on the regression list. Mark the ones with zero tolerance.
- 3Step 3: Write one pairwise judge per rule
Each judge sees the input, output A, and output B, and answers A, B, or tie. The rubric has five parts:
- The rule, in your own words.
- What better looks like and what worse looks like.
- When to call a tie: both meet the rule, both break it, or the difference would not change what the reader does. Without this, a judge invents preferences.
- The near misses that are not violations. A short proof point is not a subject line. Write them down or the judge will flag them.
- What the letter means: the answer names the better output. Say it in plain words.
Then cut what each judge sees. A judge comparing the opening of two outputs should see only the opening of each. A judge comparing one line should see only that line from each side. When the versions write in different formats, the same cut on both sides is what makes the comparison fair.
- 4Step 4: Ask twice, swap the order
Every pair runs with old as A, then with new as A.
- Same winner both times: a win.
- Tie both times: a tie.
- A winner once and a tie once: a split. It counts as a tie.
- Opposite winners: a flip. It counts as a tie and as an inconsistency. Several flips on one rule means that judge is reading position, not content.
- 5Step 5: Test the judge before you read the result
build_pairs.pybuilds three kinds of test pairs from your reviewed outputs:- Planted pairs: a reviewed output against a copy with one mistake planted by a script. The right answer is known the moment you write the edit. The judge must pick the clean side in both orders.
- Identical pairs: an output against itself. The judge must call a tie in both orders. A judge that picks a side here is guessing.
- Held out pairs: new mistake styles you add after the first run and never use to tune a rubric. A judge that catches these learned the rule. One that only catches the originals learned the test.
The bar: every planted and held out pair caught on a zero tolerance rule, 90% or more on the rest, every identical pair a tie, and the same verdict in both orders on 95% or more of pairs. Read every miss with the judge's reason before you change anything.
- 6Step 6: Read the result in this order
- The regression list. Any must-never or must-always loss, case by case, with the judge's reason.
- The other losses. Where the old version was better on a quality rule.
- The wins. Did the change do what you made it for.
- 7Step 7: Decide, then grow the set
Ship, fix and rerun, or hold. The REVIEWER decides, with the regression list in hand. After that:
- Every regression becomes a planted mistake. The next prompt change gets tested against it.
- A rubric change means a rerun. A verdict from an older rubric is stale.
- Keep the report with the change. A prompt version without its comparison is an opinion.
- Both versions ran on the same frozen inputs, same model, same settings.
- Every judge maps to exactly one rule, labeled quality, must-always, or must-never.
- Nothing a script can check is sent to a judge.
- Every rubric says when to tie and what the letter means, and lists its near misses.
- Every pair ran in both orders, and only a winner in both counts.
- The judge passed planted, identical, and held out pairs before its verdicts were read.
- Held out styles were added after the first run and never used to tune a rubric.
- The regression list was read before the win counts, by a person.
- A must-never loss was never offset by wins elsewhere.
- No em dashes anywhere.
PROMPT JUDGE REPORT the fictional sample, both orders, 60 judge calls, about 15 cents
CAN THE JUDGE BE TRUSTED?
criterion planted held out identical same verdict flips
caught caught tied both orders
decision_up_top 2 of 2 1 of 1 2 of 2 10 of 10 0
because_names_cause_and_problem 2 of 2 1 of 1 2 of 2 10 of 10 0
no_outreach_copy 2 of 2 1 of 1 2 of 2 10 of 10 0
DID THE CHANGE HELP? 5 inputs, old prompt vs new prompt
criterion kind new wins old wins ties
decision_up_top quality 5 0 0
because_names_cause_and_problem must-always 1 1 3
no_outreach_copy must-never, zero tolerance 0 1 4
MODELS
decision_up_top and because_names_cause_and_problem on Claude Haiku 4.5
no_outreach_copy on Claude Sonnet 5.5
REGRESSION LIST any loss on a must-never or must-always rule
no_outreach_copy, harbor-doors: old won in both orders.
judge: the new Open with greets Lee, speaks in the second person, and asks
for 15 minutes next week. that is outreach copy. the old one states a
customer result, which is not.
because_names_cause_and_problem, fern-electric: old won in both orders.
judge: the old Because names a dated cause (a second dispatcher hire) and
a problem (dispatch overload). the new one says Marcus "likely owns the
decision", a stand-in for a problem.
OTHER LOSSES, for a person to read
none.
DECISION
the new prompt does what it was written for: the decision is up top in all 5.
it does not ship yet. two regressions: one on a zero tolerance rule, and one
Because that lost its cause. fix the prompt, plant both as test cases, and
rerun.
Where an operator takes this next.
Run the new version against the old one and against the version before it. A change that wins against last week and loses against last month is drift.
For rules about invented facts, give the judge the pages each version read, and compare which output stays closer to them. `rag-judge` covers that check for one output.
Keep the inputs, the reviewed outputs, and the planted mistakes. The next comparison costs one command.
One skill is the on-ramp.
A single skill does one job. Chained into a playbook, or run as a full build, it becomes a system. Here is where this one plugs in.