Evaluation toolkit for RAG (Retrieval-Augmented Generation) pipelines. Measures what matters at each stage: chunk quality, retrieval quality, and answer quality — all without calling an LLM judge.
| Stage | Metrics |
|---|---|
| Chunking | Fixed-size (with overlap), paragraph boundary, sentence boundary |
| Retrieval | Context Precision, Context Recall, NDCG@K, MRR |
| Answer quality | Faithfulness (trigram overlap), Answer Relevance (Jaccard) |
| End-to-end | RagEvaluator.eval() runs all stages in one call |
pip install -e .from rageval import RagEvaluator
def my_retriever(query, docs, k):
# plug in your actual retriever
...
evaluator = RagEvaluator(retriever=my_retriever)
result = evaluator.eval(
query="What causes transformer attention to scale quadratically?",
docs=corpus,
relevant_ids=[2, 7],
answer="Attention computes pairwise dot products across all tokens, growing as O(n²) in sequence length.",
k=5,
)
print(result)query: "What causes transformer attention to scale quadratically?"
k: 5
── Retrieval ─────────────────────────────────────
context_precision: 0.8000
context_recall: 1.0000
ndcg: 0.9307
mrr: 1.0000
── Answer ────────────────────────────────────────
faithfulness: 0.8571
answer_relevance: 0.6667
from rageval.chunking import chunk_fixed, chunk_by_paragraph, chunk_by_sentence
text = open("doc.txt").read()
chunks = chunk_fixed(text, size=512, overlap=64)
chunks = chunk_by_paragraph(text, min_size=64, max_size=1024)
chunks = chunk_by_sentence(text, max_size=512)from rageval.metrics import context_precision, context_recall, ndcg_at_k, mrr
retrieved = ["doc_3", "doc_1", "doc_7", "doc_9", "doc_2"]
relevant = {"doc_1", "doc_7"}
print(context_precision(retrieved, relevant, k=5)) # 0.4
print(context_recall(retrieved, relevant)) # 1.0
print(ndcg_at_k(retrieved, relevant, k=5)) # 0.6934
print(mrr(retrieved, relevant)) # 0.5from rageval.metrics import faithfulness, answer_relevance
context = "Transformers compute attention across all token pairs, giving O(n²) complexity."
answer = "Attention scales quadratically because every token attends to every other token."
query = "Why is attention quadratic?"
print(faithfulness(answer, context)) # 0.75 — fraction of answer sentences grounded in context
print(answer_relevance(answer, query)) # 0.60 — Jaccard on non-stopword tokensNo LLM judge. Faithfulness uses trigram overlap against the retrieved context. Answer relevance uses Jaccard similarity on non-stopword tokens. Both are fast, deterministic, and need no API key — suitable for CI.
Pluggable retriever. RagEvaluator takes any (query, docs, k) → list[tuple[doc, score]] callable. Swap in BM25, a vector store, or a mock for testing.
pytest -q...........................................
43 passed in 0.08s