ZerdaZerda

RAGEvaluator, EvalSample, EvalResult, EvalReport, MetricScore

On this page

core.evaluation.RAGEvaluator and supporting dataclasses

Overview

RAGEvaluator measures RAG answer quality across five RAGAS-style axes — faithfulness, answer relevancy, context precision, context recall, and answer correctness — using an LLM as judge, blended with fast lexical-overlap heuristics as a fallback/stabilizer. It has no external dependencies (no ragas package, no embeddings library) and works with any object exposing .generate(prompt) -> str.

[!NOTE] Availability This module is imported into core's top-level namespace inside a try/except ImportError block (see Architecture → Package Initialization) — it is expected to always succeed given the module's own dependencies (all standard library), but the surrounding code defensively tolerates its absence.

Metrics computed

MetricRequires ground_truth?Meaning
faithfulnessNoIs the answer grounded in the retrieved context?
answer_relevancyNoDoes the answer actually address the question?
context_precisionNoAre the retrieved chunks genuinely relevant to the question?
context_recallYesDo the retrieved chunks cover what's needed to reach the reference answer?
answer_correctnessYesHow similar is the generated answer to the reference answer?

When ground_truth is absent, context_recall and answer_correctness are returned as MetricScore(..., skipped=True) and excluded from overall()/aggregate means.

Dataclasses

EvalSample

Python · 7 lines
@dataclass
class EvalSample:
    question:     str
    answer:       str
    contexts:     List[str]
    ground_truth: Optional[str] = None
    sample_id:    Optional[str] = None

A single evaluation unit — one question, one generated answer, the context chunks used to produce it, and (optionally) a reference answer.

MetricScore

Python · 6 lines
@dataclass
class MetricScore:
    name:      str
    score:     float
    reasoning: str = ""
    skipped:   bool = False

__repr__ returns MetricScore({name}=N/A [skipped]) when skipped, else MetricScore({name}={score:.3f}).

EvalResult

Python · 9 lines
@dataclass
class EvalResult:
    sample_id:           Optional[str]
    faithfulness:        MetricScore
    answer_relevancy:    MetricScore
    context_precision:   MetricScore
    context_recall:      MetricScore
    answer_correctness:  MetricScore
    elapsed_sec:         float = 0.0

Methods

  • scores() -> Dict[str, Optional[float]] — {metric_name: rounded_score_or_None}, None for skipped metrics.
  • overall() -> float — mean of all non-skipped metric scores, rounded to 4 decimals; 0.0 if all metrics were skipped.
  • summary() -> str — multi-line human-readable report including per-metric scores, overall, and elapsed.
  • to_dict() -> Dict[str, Any] — scores() plus overall, elapsed_sec, sample_id.

EvalReport

Python · 4 lines
@dataclass
class EvalReport:
    results:    List[EvalResult] = field(default_factory=list)
    total_time: float = 0.0

Methods

  • mean_scores() -> Dict[str, Optional[float]] — per-metric mean across all samples (None if a metric was skipped for every sample).
  • std_scores() -> Dict[str, Optional[float]] — per-metric sample standard deviation (statistics.stdev); None if fewer than 2 non-skipped values exist for that metric.
  • overall_mean() -> float — mean of each sample's overall().
  • to_dict() -> Dict[str, Any] — num_samples, total_time_s, mean_scores, std_scores, overall_mean, and per-sample samples (each via EvalResult.to_dict()).
  • print_report() -> None — prints a formatted ASCII report to stdout, including ± std where available.

RAGEvaluator

Exact signature

Python · 2 lines
class RAGEvaluator:
    def __init__(self, llm: Any, language: str = "auto", batch_delay: float = 0.0) -> None: ...

Parameters

NameTypeRequiredDefaultNotes
llmAnyYes—Must expose .generate(prompt) -> str.
languagestrNo"auto""ar", "en", or "auto" (per-sample detection).
batch_delayfloatNo0.0Seconds to sleep between samples in evaluate_dataset — a rate-limit guard.

Raises

ValueError("RAGEvaluator requires an LLM instance.") if llm is None.

_detect_lang (internal)

Returns self.language if not "auto"; otherwise "ar" if any character is in \u0600–\u06FF, else "en" — same "any character" heuristic as RAGSystem._detect_language_simple.

_call_llm (internal)

Wraps self.llm.generate(prompt) in a try/except; returns "" (and logs a warning) on any failure, rather than propagating.

Metric scoring pattern

All five _faithfulness / _answer_relevancy / _context_precision / _context_recall / _answer_correctness internal methods follow the same pattern:

  1. Compute a fast lexical heuristic score via _tok_overlap (Jaccard token overlap between two strings, language-agnostic — re.findall(r"\w+", text.lower()) on each side).
  2. Format a metric-specific bilingual prompt template (see source for the exact Arabic/English wording) and call the LLM, requesting a JSON response {"score": 0.0-1.0, "reasoning": "..."}.
  3. Parse the response with _parse_score(text, default=heuristic).
  4. Blend: blended = round(0.7 * llm_score + 0.3 * heuristic, 4).
  5. If the LLM call returned an empty string (failure), skip the blend and return the heuristic score directly with reasoning "LLM unavailable — heuristic used.".

Each metric has its own early-exit "skipped"/zero-score conditions (e.g. faithfulness is skipped=True with score 0.0 if the answer is empty or there are no contexts; context_recall/answer_correctness are skipped=True if ground_truth is None).

_parse_score (module-level helper)

Python
def _parse_score(text: str, default: float = 0.5) -> Tuple[float, str]
  1. Strip ```json / ``` fences and attempt json.loads; if successful, extract score (clamped to [0, 1]) and reasoning.
  2. On JSON failure, fall back to a regex search for the first float-like token matching 0.xxx, 1.0, 0, or 1 in the raw text.
  3. If nothing parseable is found, return (default, text.strip()).

evaluate

Python
def evaluate(self, sample: EvalSample) -> EvalResult

Computes all five metrics for one sample (via the internal _* methods above), measures wall-clock time with time.perf_counter(), and returns a populated EvalResult. Logs an INFO summary line with overall() and elapsed time.

Raises

None explicitly — all LLM failures are absorbed into heuristic fallbacks via _call_llm.

Examples

Basic:

Python · 11 lines
from zerda.rag.core import RAGEvaluator, EvalSample

evaluator = RAGEvaluator(llm=my_llm)
sample = EvalSample(
    question="ما عاصمة مصر؟",
    answer="عاصمة مصر هي القاهرة.",
    contexts=["القاهرة هي عاصمة مصر وأكبر مدنها."],
    ground_truth="القاهرة",
)
result = evaluator.evaluate(sample)
print(result.summary())

Without ground truth (two metrics skipped):

Python · 5 lines
sample = EvalSample(question="...", answer="...", contexts=["..."])
result = evaluator.evaluate(sample)
print(result.scores())
# {'faithfulness': 0.81, 'answer_relevancy': 0.76, 'context_precision': 0.65,
#  'context_recall': None, 'answer_correctness': None}

evaluate_dataset

Python
def evaluate_dataset(self, samples: List[EvalSample], show_progress: bool = False) -> EvalReport

Evaluates each sample sequentially (not in parallel — unlike Reranker's LLM scoring, there is no thread pool here), optionally printing a \r-overwriting progress line, optionally sleeping batch_delay seconds between samples. Returns an aggregated EvalReport.

Empty input: logs a warning and returns an empty EvalReport() (with results=[], total_time=0.0).

Complexity

O(n) LLM calls (5 per sample), executed strictly sequentially — total wall-clock time ≈ n × (5 × LLM latency + batch_delay).

Examples

Basic batch evaluation:

Python · 6 lines
samples = [
    EvalSample(question=q, answer=a, contexts=c, ground_truth=gt)
    for q, a, c, gt in my_qa_pairs
]
report = evaluator.evaluate_dataset(samples, show_progress=True)
report.print_report()

Rate-limited batch:

Python · 3 lines
evaluator = RAGEvaluator(llm=my_llm, batch_delay=0.5)  # 0.5s between samples
report = evaluator.evaluate_dataset(samples)
print(report.to_dict())

quick_eval (classmethod)

Python · 5 lines
@classmethod
def quick_eval(
    cls, llm: Any, question: str, answer: str, contexts: List[str],
    ground_truth: Optional[str] = None,
) -> EvalResult

One-liner convenience: constructs an evaluator and an EvalSample internally, evaluates, and returns the EvalResult. Equivalent to RAGEvaluator(llm=llm).evaluate(EvalSample(...)).

Python · 8 lines
result = RAGEvaluator.quick_eval(
    llm=my_llm,
    question="ما عاصمة مصر؟",
    answer="القاهرة هي عاصمة مصر.",
    contexts=["القاهرة عاصمة مصر وأكبر مدنها."],
    ground_truth="القاهرة",
)
print(result.summary())

Thread safety

RAGEvaluator holds no shared mutable state beyond constructor arguments; evaluate_dataset is intentionally sequential, so no additional synchronization is needed for a single evaluator instance used from one thread.

Side effects

  • Calls self.llm.generate() up to 5 times per evaluate() call.
  • evaluate_dataset(show_progress=True) writes directly to stdout via print(..., end="\r").
  • EvalReport.print_report() writes a formatted report to stdout.
  • Logs at DEBUG/INFO/WARNING levels.

Best practices

  • Always supply ground_truth when you can — two of the five metrics are skipped without it.
  • Use batch_delay when evaluating against a rate-limited LLM API.
  • Treat the 70/30 LLM/heuristic blend as a stabilizer, not a guarantee of accuracy — heuristic token overlap is a crude proxy and can disagree substantially with the LLM judge for paraphrased (non-lexically-overlapping) correct answers.

Common mistakes

  • Assuming evaluate_dataset parallelizes LLM calls the way Reranker does — it does not; it's strictly sequential.
  • Forgetting that overall() and aggregate means silently exclude skipped metrics rather than treating them as 0.0 — a sample with no ground_truth is scored purely on the three ground-truth-free metrics.
  • RAGSystem — typically the source of the answer/contexts used to build EvalSamples.

Related pages