Find out whether the new prompt is actually better.
“Did my prompt change actually help, or did I just break something I cannot see?”
Runs the old and new prompt on the same inputs and judges each pair in both orders, counting only wins that hold both ways, with a list of anything the new version broke.
What this skill does
You changed the prompt. The new outputs look better. You read five of them and they are tighter, clearer, more useful.
Then it ships, and a week later someone finds the one thing the old version never did. The rewrite that fixed the format also started slipping a ready-to-send email into the brief. Nobody compared for that, because everybody was comparing for the thing they set out to fix.
This skill runs the old prompt and the new prompt on the same inputs and puts the outputs side by side in front of a Claude judge, one rule at a time. For each input and each rule, the judge answers A, B, or tie. Then it asks again with the two outputs swapped, because judges tend to prefer whichever one they read first. A winner only counts when it wins both times.
You get two answers. Did the change help, rule by rule. And a regression list: every case where the new version is worse on a rule it must never break or must always keep, even if it wins everywhere else.
A judge is a prompt. Before its verdicts count, it gets tested: pairs where you planted the mistake yourself, and pairs where both sides are the same output. If it cannot pick the clean side every time, or cannot call a tie on two copies of the same thing, it is not ready to referee your prompt.
How this differs from brief-prompt-skill: that skill grades one prompt on its own, against a checklist, and compares its pass rate to a naive one-line baseline. This one puts two real versions head to head on the same inputs, asks in both orders, tests the judge first, and lists regressions. Use brief-prompt-skill to write a prompt and get its pass rate. Use this one when you are about to replace a prompt that already works. It builds on agent-judge, which grades one output against one rule, pass or fail.
Inputs
- Works today with: saved outputs from both versions on the same inputs, named so the same input has the same file name in both folders. Ten inputs is a start. Thirty is better.
- A few outputs you trust: outputs a person already reviewed. They can come from either version. The trust test plants mistakes into copies of them.
- Your rules, written down: the ones a script cannot check. If you have never written them, run
agent-contractfirst. - An Anthropic API key: the judges are Claude models called through the Anthropic SDK.
- Python: the starter scripts install Ragas, pinned to versions that work together.
How it works
- 1Step 1: Run both versions on the same inputs
Freeze the inputs. Run the old prompt and the new prompt on the same model with the same settings, and save each output under its input's name. If one version stops on an input where the other answers (asks a question, refuses, returns nothing), that is a finding a script can see. List it, do not judge it.
- 2Step 2: Split the rules
Anything checkable by shape (word counts, required lines, valid links, a banned string) goes to a script and runs on both sides first. Only rules that need meaning get a judge. Label each one:
- quality: better is better, a loss is worth reading.
- must-always: a loss goes on the regression list. Losing a rule the old version kept is a regression, even when the new version wins elsewhere.
- must-never: a loss goes on the regression list. Mark the ones with zero tolerance.
- 3Step 3: Write one pairwise judge per rule
Each judge sees the input, output A, and output B, and answers A, B, or tie. The rubric has five parts:
- The rule, in your own words.
- What better looks like and what worse looks like.
- When to call a tie: both meet the rule, both break it, or the difference would not change what the reader does. Without this, a judge invents preferences.
- The near misses that are not violations. A short proof point is not a subject line. Write them down or the judge will flag them.
- What the letter means: the answer names the better output. Say it in plain words.
Then cut what each judge sees. A judge comparing the opening of two outputs should see only the opening of each. A judge comparing one line should see only that line from each side. When the versions write in different formats, the same cut on both sides is what makes the comparison fair.
- 4Step 4: Ask twice, swap the order
Every pair runs with old as A, then with new as A.
- Same winner both times: a win.
- Tie both times: a tie.
- A winner once and a tie once: a split. It counts as a tie.
- Opposite winners: a flip. It counts as a tie and as an inconsistency. Several flips on one rule means that judge is reading position, not content.
- 5Step 5: Test the judge before you read the result
build_pairs.pybuilds three kinds of test pairs from your reviewed outputs:- Planted pairs: a reviewed output against a copy with one mistake planted by a script. The right answer is known the moment you write the edit. The judge must pick the clean side in both orders.
- Identical pairs: an output against itself. The judge must call a tie in both orders. A judge that picks a side here is guessing.
- Held out pairs: new mistake styles you add after the first run and never use to tune a rubric. A judge that catches these learned the rule. One that only catches the originals learned the test.
The bar: every planted and held out pair caught on a zero tolerance rule, 90% or more on the rest, every identical pair a tie, and the same verdict in both orders on 95% or more of pairs. Read every miss with the judge's reason before you change anything.
- 6Step 6: Read the result in this order
- The regression list. Any must-never or must-always loss, case by case, with the judge's reason.
- The other losses. Where the old version was better on a quality rule.
- The wins. Did the change do what you made it for.
- 7Step 7: Decide, then grow the set
Ship, fix and rerun, or hold. The REVIEWER decides, with the regression list in hand. After that:
- Every regression becomes a planted mistake. The next prompt change gets tested against it.
- A rubric change means a rerun. A verdict from an older rubric is stale.
- Keep the report with the change. A prompt version without its comparison is an opinion.
Output format
The report from the fictional sample that ships with this skill, as it came back on the run before publishing.
PROMPT JUDGE REPORT the fictional sample, both orders, 60 judge calls, about 15 cents
CAN THE JUDGE BE TRUSTED?
criterion planted held out identical same verdict flips
caught caught tied both orders
decision_up_top 2 of 2 1 of 1 2 of 2 10 of 10 0
because_names_cause_and_problem 2 of 2 1 of 1 2 of 2 10 of 10 0
no_outreach_copy 2 of 2 1 of 1 2 of 2 10 of 10 0
DID THE CHANGE HELP? 5 inputs, old prompt vs new prompt
criterion kind new wins old wins ties
decision_up_top quality 5 0 0
because_names_cause_and_problem must-always 1 1 3
no_outreach_copy must-never, zero tolerance 0 1 4
MODELS
decision_up_top and because_names_cause_and_problem on Claude Haiku 4.5
no_outreach_copy on Claude Sonnet 5.5
REGRESSION LIST any loss on a must-never or must-always rule
no_outreach_copy, harbor-doors: old won in both orders.
judge: the new Open with greets Lee, speaks in the second person, and asks
for 15 minutes next week. that is outreach copy. the old one states a
customer result, which is not.
because_names_cause_and_problem, fern-electric: old won in both orders.
judge: the old Because names a dated cause (a second dispatcher hire) and
a problem (dispatch overload). the new one says Marcus "likely owns the
decision", a stand-in for a problem.
OTHER LOSSES, for a person to read
none.
DECISION
the new prompt does what it was written for: the decision is up top in all 5.
it does not ship yet. two regressions: one on a zero tolerance rule, and one
Because that lost its cause. fix the prompt, plant both as test cases, and
rerun.
Tips
- Both versions ran on the same frozen inputs, same model, same settings.
- Every judge maps to exactly one rule, labeled quality, must-always, or must-never.
- Nothing a script can check is sent to a judge.
- Every rubric says when to tie and what the letter means, and lists its near misses.
- Every pair ran in both orders, and only a winner in both counts.
- The judge passed planted, identical, and held out pairs before its verdicts were read.
- Held out styles were added after the first run and never used to tune a rubric.
- The regression list was read before the win counts, by a person.
- A must-never loss was never offset by wins elsewhere.
- No em dashes anywhere.
Make the AI answer from your documents, and say when it can't.
What is RAG, and do I need it?
Find out whether it was the search or the answer.
Is our AI making things up?