Turn every thumbs down into a test that catches it next time.
Sorts each piece of feedback into a broken rule, a gap in your rules, or a preference, then plants the mistake into good outputs and proves a check catches it before it joins your gate.
In plain English: “How do I stop the same mistake coming back?”
Inside: The feedback ledger, planters, script checks and judges, and a fictional sample with six pieces of feedback.
Install it in one line, or paste it in.
~/.claude/skills/ and runs automatically when it is relevant.Connect your context. Set it to your motion.
a pile of real feedback (thumbs down with reasons, review notes, rejected changes), the outputs that feedback was about if you saved them, and 10 or more outputs you trust.
a contract, a system prompt, or a list. Feedback is classed against them. If you have none, run `agent-contract` first.
the judges are Claude models called through Ragas and the Anthropic SDK. Script checks need neither.
it covers how to write and trust-test a judge in depth. This skill decides which feedback deserves one.
RULE_OWNER does not move. A drafted rule is a proposal. Until a person approves it, nothing is gated on it.
| Set this | What it is | Default / Example |
|---|---|---|
| FEEDBACK_SOURCE | where feedback comes from | thumbs down with a reasonreview notesrejected or reverted changes |
| CONTRACT | the written rules feedback is classed against | your contractelse the system prompt |
| CLEAN_SET | outputs that passed a review | 10 or morefrom the current version |
| STYLES | planter styles per piece of feedback | 2 to 3 from the feedback's own wording |
| HOLDOUT | new styles added after the first run, never used to tune | 2 per rule |
| JUDGE_MODEL | the model each judge runs on | start on Claude Haiku 4.5move up only when a rerun shows it |
| REPEATS | runs per judgment | 2 to tune5 for a gate |
| GATE_BAR | what a check must clear to join the gate | every plant caught on a zero tolerance rule90% or more otherwisefalse alarms under 5% |
| PREFERENCE_BAR | when a logged preference becomes a contract question | the same preference from 3 different people |
| RULE_OWNER | who approves drafted rules | the agent's owner. never the skill |
Everything the skill does, in full.
Your eval set was written before anyone used the agent. Your users have been writing the real one since.
Every thumbs down with a reason is a test case someone handed you for free. Most teams fix the prompt, maybe rerun that one input, and move on. The next version breaks the same way on a different input and nobody notices until another thumbs down.
This skill takes each piece of feedback and asks three questions before it builds anything:
- Which rule did it break? Map it to a line in your written rules. If no line covers it, that is a gap in the rules, and the skill drafts the missing line for a person to approve. It never adds one quietly.
- How do I make many of these? A planter: a small scripted edit that puts the same kind of mistake into outputs you already trust. One complaint becomes ten or twenty cases.
- What catches it? A script check if a regex can settle it. A Claude judge if it needs meaning. An existing judge if one already covers the rule.
Then the proof: the check has to catch every planted copy and pass every clean output before it counts toward your gate.
Taste is not a rule. "I'd prefer paragraphs" gets logged, not gated. A gate built on one person's taste blocks good outputs forever.
Built in October 2026 on the real feedback log of a research agent that had been live for a few weeks: thumbs down from the people using it, notes from a reviewer, and changes its owner approved, then later reverted. The agent writes a brief a person reads before making a call.
The log was more useful than the eval set written before launch, and it taught the method:
- The fixes went into the prompt and stopped there. Several rules that came out of feedback lived in the system prompt and the skills, but never made it back into the written contract. Classing each item against the contract is what surfaced the drift.
- Some feedback was later overruled. A reviewer flagged a missing check, it became a rule, and the owner removed it a few versions later. Built as a gate, it would have failed outputs the owner wanted. Feedback that the owner overrules is a preference.
- The saved outputs were older than the rules. The "clean" outputs were saved before some rules existed, so they failed checks written from later feedback. Those flags were real misses in stale outputs, not broken checks.
- The planters were wrong more often than the judges. When a judge missed a plant, the plant was usually the problem: an edit that left an attribution in place, or landed in a line that already said the right thing. Read one plant of each style before you pay for a run.
- Feedback without the output is half a test case. The live runs saved the verdict and the comment, not the output. Those items could only be reproduced from their description, and some could not be tested at all.
What is still unsolved: a judge sees the output, not what the agent should have found. Feedback like "it named the wrong decision maker" needs the transcript or source passed in next to the output. That part is not in this kit yet.
A fictional sample ships with this skill: four call recaps, six pieces of feedback on them in example/feedback.md, and the judges and checks that feedback turned into.
- Copy the
scripts/andexample/folders from this skill somewhere you can run Python 3.10 or newer. - Install and set your key:
python3 -m venv .venv && .venv/bin/pip install -r scripts/requirements.txt export ANTHROPIC_API_KEY=... - Read
example/feedback.mdfirst. It is the ledger: each piece of feedback, its class, and what it turned into. - Test the script checks (free), then build the test set and run the judges:
.venv/bin/python scripts/run_checks.py --judges example/judges.py --clean example/outputs --out feedback-run .venv/bin/python scripts/build_set.py --judges example/judges.py --clean example/outputs --out feedback-run .venv/bin/python scripts/run_judges.py --judges example/judges.py --out feedback-run - Open
feedback-run/checks-report.mdandfeedback-run/results/<time>/report.md. Read the tables, then every disagreement under them.
The sample is about 40 judge calls and costs a few cents.
Your first exercise is already in the report. On the run before this was published, the commitment judge failed the Oakmoor recap, the one clean recap where the customer really did commit ("The budget is approved. Anything under it, I sign."). The rubric names that exact near miss as a pass, and the judge still read it as conditional. Decide who is right. If the judge is wrong, try narrowing what it sees or moving it up a model, and rerun. If the recap overstated it, fix the recap. That loop is the whole skill in miniature.
- 1Step 1: Collect feedback with its output
Pull every piece of feedback with the output it was about, the version that produced it, and who said it. Feedback with no saved output can still be classed, but it can only be reproduced from its description. Note which items are missing their output.
- 2Step 2: Class every item
One row per item in a ledger (
example/feedback.mdshows the format).Class Test Becomes Rule broken a written rule already says the output was wrong a planter, plus a script check or a judge Gap no written rule covers it, or it lives in the prompt but not the contract a drafted rule for RULE_OWNER. no gate until approved Preference taste, a one-off, or something the owner later overruled a line in the ledger. never a gate Two traps. Feedback that the agent followed a bad rule is a gap, not a broken rule: the fix is the rule. And a reviewer's note is not automatically a rule: check whether the owner kept it.
- 3Step 3: Script or judge
If a regex can settle it (a missing field, a word cap, a banned character, a number next to a word), it goes to a script check. Free, exact, and it runs first. If it needs meaning (was this commitment real, is this addressed to the customer), it goes to a judge. If a judge already covers the rule, the feedback becomes a new planter style for that judge, not a new judge.
- 4Step 4: Write the planters
A planter takes one clean output and returns it with the same mistake the feedback named, or nothing if it cannot apply. Write 2 to 3 styles from the feedback's own wording. Make the planter refuse outputs where the edit would be true: planting "confirmed the budget" on a recap where the customer really did confirm it is not a mistake.
Read one plant of each style before the first paid run. Planters are code, and code has bugs.
- 5Step 5: Prove it
Run the clean outputs and the plants, twice. Then add held-out styles the rubric never saw and run again. Read every disagreement. There are four kinds:
- The judge or check is wrong. Fix the rubric, the view, or the regex, and rerun.
- The clean output really breaks the rule. Often because the output is older than the rule. Count it as a catch.
- The rule is vague. RULE_OWNER decides, the rule is rewritten, the judge follows.
- The plant is wrong. Fix the planter.
- 6Step 6: Join the gate, keep the ledger
A check joins the gate once it clears GATE_BAR. Its row in the ledger names the feedback it came from, so anyone can trace a gate back to the person who asked for it. Each new batch of feedback starts at Step 1, and the set grows. It never restarts.
- Every piece of feedback has a ledger row with a class.
- No drafted rule is gated until a person approves it.
- No preference became a gate.
- Every rule broken maps to a named rule.
- Nothing a regex can check went to a judge.
- Every planter was read on at least one output before a paid run.
- Every check was run on clean, planted, and held-out cases before it joined the gate.
- Held-out styles were added after the first run and never used to tune.
- Every disagreement was read and sorted into one of the four kinds.
- The report names what could not be tested and what is missing.
- No em dashes anywhere.
FEEDBACK LEDGER 6 items: 4 rule broken, 1 gap, 1 preference
FB1 "says she agreed to the three year term" rule broken, line 3 new judge
FB2 "next steps are a ready to send email" rule broken, line 4 existing judge, new planter
FB3 "next step has nobody on it" rule broken, line 2 script check
FB4 "missed the six week legal review" gap rule drafted, waiting
FB5 "too many bullets" preference logged
FB6 "named the wrong decision maker" rule broken, line 3 needs the transcript. open
SCRIPT CHECKS free
check false alarms planted held out
owner_and_date 0 of 4 2 of 2 2 of 2
no_dashes 0 of 4 2 of 2 2 of 2
JUDGES 2 repeats, 36 judge calls, about 5 cents, Claude Haiku 4.5
judge false alarms planted held out regex caught consistency
no_overstated_commitment 1 of 4 4 of 4 2 of 2 5 of 6 1.00
written_to_the_rep 0 of 4 2 of 2 2 of 2 0 of 4 1.00
DISAGREEMENTS, for a person to read
no_overstated_commitment, clean oakmoor-property: judge fail.
the customer said "the budget is approved, I sign". the rubric lists this as a pass.
open: judge too strict, or recap too firm? decide before this judge joins the gate.
GATE
joins now: owner_and_date, no_dashes, written_to_the_rep (with the new FB2 planter)
waiting: no_overstated_commitment (one false alarm to settle)
waiting on a person: the FB4 rule draft
not testable yet: FB6, until the transcript is passed in next to the recap
Where an operator takes this next.
Weekly feedback, weekly ledger rows. The gate grows with the agent.
If your thumbs down button stores only the reason, change it to store the output too.
If something proposes fixes from feedback, every approved fix should arrive with its planter and check, and every rejected one should land in the ledger as a preference so it is not proposed again.
For feedback about what the agent should have found, give the judge the transcript or pages next to the output.
One skill is the on-ramp.
A single skill does one job. Chained into a playbook, or run as a full build, it becomes a system. Here is where this one plugs in.