RAG Evaluation Kit

Your retrieval-augmented chatbot answers from your documents, until it doesn't: the right page isn't retrieved, the answer invents a number, or it confidently answers a question your docs never cover. This kit measures all three on a set of test questions, so you can change the chunk size, embedding model, prompt or LLM and see whether things got better or worse.

• Retrieval metrics: hit@k, recall@k, precision@k, MRR and nDCG@k against the documents (or chunks) that should be found. Chunk ids like refunds.md#3 count as the document refunds.md.
• Answer metrics: token F1, ROUGE-L and exact match against a reference answer, plus must_include facts ("5-7 business days").
• Faithfulness without an LLM: every answer sentence must be covered by a retrieved chunk, and every number it states must appear in the chunks. Catches invented timelines, prices and limits.
• Context recall: did the retrieved chunks contain the reference answer at all?
• Abstention: mark questions your docs don't cover with "answerable": false; the bot should say it doesn't know.
• Optional LLM judge (--judge openai:<model>, anthropic:<model>, OpenRouter or local Ollama): faithfulness, relevance and correctness scores with a reason.
• BM25 baseline + chunk-size sweep over a folder of .md, .txt or .html docs: a number your vector search has to beat, and the chunk size that works for your docs.
• Any pipeline: ragcheck run --pipeline "python3 my_rag.py" (JSON in, JSON out, any language), or score a JSONL file you already logged (LangChain, LlamaIndex, your own code).
• Compare runs and CI gates (--min faithfulness=0.9 --min hit@5=0.8, exit 1 below). Text, Markdown and JSON reports.
• Python 3.9+ standard library only. Claude Code skill included (skills/rag-eval).

python3 -m ragcheck bm25 qa.jsonl --docs docs/ --sweep 100,200,400          # baseline + chunk sizes
python3 -m ragcheck run qa.jsonl --pipeline "python3 my_rag.py" --out run-a.json
python3 -m ragcheck eval qa.jsonl logged-results.jsonl --judge openai:<model> --out run-b.json
python3 -m ragcheck compare run-a.json run-b.json
python3 -m ragcheck eval qa.jsonl results.jsonl --min faithfulness=0.9 --min hit@5=0.8 --format md >> "$GITHUB_STEP_SUMMARY"

./demo.sh evaluates a fictional coffee shop's help-center bot (8 docs, 11 questions): the chunk-size sweep, then a recorded LLM run where ragcheck catches a retrieval miss ("torn open" vs "damaged"), an invented "10 business days" refund time and an answer about a café that doesn't exist. ./run_tests.sh runs 8 test groups.

Dataset (qa.jsonl), one question per line:

{"id": "q05", "question": "How quickly do refunds reach my card?", "ground_truth": "Within 5-7 business days after approval.", "relevant": ["refunds.md"], "must_include": ["5-7 business days"]}
{"id": "q11", "question": "Do you have a café in Denver?", "answerable": false}

Results (results.jsonl, or what your --pipeline prints for each question):

{"id": "q05", "answer": "Refunds arrive within 5-7 business days.", "contexts": [{"id": "refunds.md#2", "text": "Refunds go back to..."}]}

contexts can also be plain strings; text, content and page_content keys are all read. The pipeline gets {"id": ..., "question": ...} on stdin and prints one JSON line. examples/simple_rag.py is a short example (BM25 + extractive answers).

• Low hit@k / recall@k: fix retrieval first (chunking, hybrid search, query rewriting); the LLM can't use what it never sees.
• Good retrieval, low context recall: chunks are found but cut off the answer; try larger chunks or more overlap.
• Low faithfulness: the model adds facts not in the context; tighten the prompt ("answer only from the context, say you don't know otherwise").
• Low answer F1 with high faithfulness: answers are correct but worded differently; check must_include and the judge before worrying.

• The built-in faithfulness check is word overlap, not understanding: it catches invented numbers and off-context sentences, but a claim built from words that all appear in the context (e.g. "free express shipping" when only standard shipping is free) can pass. Use --judge for those, and spot-read the flagged questions.
• Scores are only as good as your test questions: 30-100 real user questions with references beat thousands of generated ones.
• No embedding model is included; BM25 is the baseline. Your own retriever runs through --pipeline or logged results.