Make the AI answer from your documents, and say when it can't.
Builds a small RAG over your own documents: finds the passages that match a question and answers from them with a citation on every sentence, or says the answer is not there.
In plain English: “What is RAG, and do I need it?”
Inside: A readable pipeline with no vector database, a fictional sales library to try it on, and when to just paste your docs in instead.
Install it in one line, or paste it in.
~/.claude/skills/ and runs automatically when it is relevant.Connect your context. Set it to your motion.
a folder of your documents as `.md` or `.txt` files. Export from Notion, Google Docs, or a wiki to Markdown first.
only the answer step calls Claude. Indexing and search are plain Python.
it grades whether retrieval found the right passage and whether each answer stuck to it.
NOT_FOUND does not move. A RAG that never declines is a RAG that guesses, and its guesses carry your company's name.
| Set this | What it is | Default / Example |
|---|---|---|
| DOCS | the folder of documents to answer from | your playbookpricingpoliciesbattlecardsas Markdown |
| CHUNK_WORDS | how big each passage is | 220 wordssplit on headings first |
| TOP_K | how many passages the model is shown per question | 4 |
| MODEL | the model that writes the answer | claude-opus-5-5 |
| NOT_FOUND | the exact words for "the answer is not here" | Not in the sources. |
| QUESTIONS | real questions with the phrase that answers each | 10 or moreplus a few with no answer in the docs |
| REVIEWER | who reads the answers before anyone relies on them | the owner of the documents |
Everything the skill does, in full.
Ask a chatbot what your discount policy is and it will tell you. Confidently. With a number nobody at your company ever approved.
The model is answering from memory, and it has never seen your discount policy. RAG fixes that by changing the exam. Retrieval-augmented generation means: before the model answers, look up the passages in your documents that hold the answer, hand it those passages, and tell it to answer from them only, citing each one. Closed-book becomes open-book. And an open-book student who cannot find the answer in the book should say so, which is the part most RAG builds forget.
This skill teaches it by building one. A small pipeline you can read in one sitting splits your documents into passages, finds the ones that match a question, and has Claude answer from them with a citation on every sentence, or reply "Not in the sources." Then it saves each answer next to the passages it was shown, so rag-judge can grade whether the retrieval found the right passage and whether the answer stuck to it.
RAG is a retrieval problem first. The model can only answer from what the search hands it.
Built to answer one real question: "which of my skills does X, and what does it do?", asked against an operator's own catalog of published skills and posts. A few hundred documents, too many to paste into every question, and they change every week.
What the build taught:
- Finding the right document is not finding the right passage. One answer came back "Not in the sources" even though the right document was retrieved. The fact sat in a table row, and the chunk holding that row never made the top results. The decline was honest. The search was the miss.
- Keyword search went further than expected. Plain keyword ranking, the kind search engines used for decades, held up well on documents written in a consistent voice. Embeddings help with synonyms. They were not the first thing missing.
- Declining is a feature. Every question with no answer in the documents got "Not in the sources", including one trap where a document mentioned a number in passing that had nothing to do with the question.
- Your reference answers can be wrong too. One question had two correct answers in the same document, and the model quoted the other one. Write test questions carefully, or the grader will punish a right answer.
What it does not do yet: catch synonyms ("cost" will not find "price"), keep a table row attached to its heading when a chunk splits, or rank by anything but word overlap. The Go further section names the fixes.
A fictional sales library ships with this skill: five short documents for a made-up scheduling software company (pricing, discounts, ideal customer, objections, competitors) and eleven questions, three of which have no answer in the documents.
- Copy the
scripts/andexample/folders from this skill somewhere you can run Python 3.10 or newer. - Install and set your key:
python3 -m venv .venv && .venv/bin/pip install -r scripts/requirements.txt export ANTHROPIC_API_KEY=... - Index the library and ask it something:
.venv/bin/python scripts/rag.py index --docs example/docs --out rag-run .venv/bin/python scripts/rag.py ask --out rag-run "How much discount can a rep give without asking anyone?" - Run all eleven questions and look at what comes back:
Open any file in.venv/bin/python scripts/rag.py batch --out rag-run --questions example/questions.jsonlrag-run/answers/. Each one shows the question, the passages the model was shown, and its answer. Ask yourself whether every sentence in the answer is in those passages. That question is whatrag-judgeautomates.
The whole sample costs about a cent.
- 1Step 1: Decide whether you need RAG at all
Three options, simplest first:
- Paste the documents in. If everything fits in one prompt and rarely changes, put it all in the prompt and skip retrieval entirely. Current Claude models read very long prompts. This is the simplest RAG there is, and often the right one.
- RAG. When there are too many documents to paste every time, when they change often, or when every answer needs a citation someone can check.
- Fine-tuning. Almost never for facts. Training a model on your pricing teaches it to sound like your pricing, not to know this quarter's price.
- 2Step 2: Split the documents into passages
Each passage should answer one kind of question on its own. Split on headings first, then paragraphs, and keep each passage's heading with it, because the heading is often the words a question uses. Around 200 words is a reasonable start. Too small and a passage loses its context. Too big and the model is shown a page to find one line.
Write your documents for retrieval as much as for reading: one topic per section, a heading that says what the section answers, and no key fact stranded in a table with no words around it.
- 3Step 3: Find the passages that match the question
The kit uses BM25, a keyword ranking: a passage scores high when it contains the question's rarer words. No embeddings, no vector database, nothing to host. It misses synonyms and paraphrases. Add embeddings when your test set shows keyword search missing passages it should find, not before.
- 4Step 4: Answer from the passages only
The instructions do three jobs: answer only from the passages shown, cite the passage behind every sentence, and say the exact NOT_FOUND line when the passages do not hold the answer. The citation is what lets a reader, and a judge, check the answer.
- 5Step 5: Save what the model saw
Save each answer next to the question and the passages it was shown. Without that record nobody can tell a retrieval miss from an answer that drifted, and those need different fixes.
- 6Step 6: Grade it before anyone relies on it
Run
rag-judgeon the saved answers. A script checks whether the passage holding the answer was found. Two trust-tested judges check whether every claim is supported by the passages, and whether the answer declined exactly when it should have.
- Every answer cites a passage for every sentence, or says the NOT_FOUND line exactly.
- Every answer is saved with the question and the passages it was shown.
- The test set includes questions with no answer in the documents, and every one of them was declined.
- Retrieval misses are counted at the passage level, not the document level.
- Nobody relies on the answers until a person has read them and
rag-judgehas graded them. - No em dashes anywhere.
One answer from the fictional library, as `rag.py ask` returns it. ```
How much discount can a rep give without asking anyone?
[discount-policy#1] Who can approve what A rep can offer up to 10% off the annual price on their own. Between 10% and 20% needs the sales manager. Anything above 20% needs the head of sales, and comes with a written reason in the deal record.
[discount-policy#2] What we never discount Implementation for the Scale plan is never discounted, because a rushed rollout is the top reason Scale customers churn in their first year. ...
A rep can offer up to 10% off the annual price on their own [discount-policy#1]. Discounts between 10% and 20% need the sales manager's approval, and anything above 20% needs the head of sales plus a written reason in the deal record [discount-policy#1].
Implementation for the Scale plan can never be discounted [discount-policy#2].
And the batch run's progress lines on the sample:
q01: passage found | answered ... q08: passage found | answered u01: no answer expected | declined u02: no answer expected | declined u03: no answer expected | declined 11 answers -> rag-run/answers.
Where an operator takes this next.
If your test set shows passages missed because the question used different words, add embedding search alongside the keyword ranking, and keep both.
Turn table rows into sentences ("STALE_DAYS: days since last engagement, default 90") so a row survives being split from its heading.
Tag documents by product, region, or date, and filter on the tag first. Last year's pricing should not compete with this year's.
A new chunk size or a new document set can quietly break answers that used to work. Rerun the test set and `rag-judge` after each change.
One skill is the on-ramp.
A single skill does one job. Chained into a playbook, or run as a full build, it becomes a system. Here is where this one plugs in.