ZerdaZerda

Cookbook: Building an Evaluation Pipeline

On this page

Using RAGEvaluator to score a RAGSystem's answers against a labeled test set.

Recipe: Evaluate a fixed question set with ground truth

Python · 23 lines
from zerda.rag.core import RAGEvaluator, EvalSample

test_set = [
    {"question": "What is the return window?",
     "ground_truth": "30 days from purchase."},
    {"question": "Can I return sale items?",
     "ground_truth": "No, sale items are final."},
]

samples = []
for item in test_set:
    retrieved = rag.retrieve(item["question"])
    answer = rag.generate(item["question"])
    samples.append(EvalSample(
        question=item["question"],
        answer=answer,
        contexts=[chunk.text for chunk, _score in retrieved],
        ground_truth=item["ground_truth"],
    ))

evaluator = RAGEvaluator(llm=judge_llm)   # can be the same or a different LLM
report = evaluator.evaluate_dataset(samples, show_progress=True)
report.print_report()

Recipe: Regression-testing a config change

Python · 11 lines
def score_config(config):
    rag = RAGSystem(vector_db=vector_db, llm=llm, chunker=chunker,
                     context_manager=context_manager, config=config)
    rag.add_documents(corpus)
    samples = [build_sample(rag, q) for q in test_questions]
    return evaluator.evaluate_dataset(samples).overall_mean()

baseline_score = score_config(RAGConfig())
reranked_score = score_config(RAGConfig(enable_reranking=True, rerank_mode="hybrid"))

print(f"Baseline: {baseline_score:.3f}  |  Reranked: {reranked_score:.3f}")

Recipe: Exporting results for a dashboard

Python · 5 lines
import json

report = evaluator.evaluate_dataset(samples)
with open("eval_report.json", "w") as f:
    json.dump(report.to_dict(), f, indent=2, ensure_ascii=False)

Recipe: Quick single-sample sanity check during development

Python · 9 lines
from zerda.rag.core import RAGEvaluator

result = RAGEvaluator.quick_eval(
    llm=judge_llm,
    question="What is the refund policy?",
    answer=rag.ask("What is the refund policy?"),
    contexts=[c.text for c, _ in rag.retrieve("What is the refund policy?")],
)
print(result.summary())

Recipe: Rate-limiting evaluation against a throttled API

Python · 2 lines
evaluator = RAGEvaluator(llm=judge_llm, batch_delay=1.0)  # 1s pause between samples
report = evaluator.evaluate_dataset(large_sample_set)

Remember evaluate_dataset runs strictly sequentially (no thread pool), so batch_delay combined with a large sample set and 5 LLM calls per sample can add up — budget wall-clock time accordingly (see evaluate_dataset complexity).

Related pages