Tutorial: A/B Testing Prompt Variants

1. Define variants

from zerda.prompt.ab_testing import PromptVariant

concise = PromptVariant(
    name="concise",
    template="Context: {context}\n\nQ: {question}\nA (brief):",
)

grounded = PromptVariant(
    name="grounded",
    template=(
        "Answer strictly using the context below. If unsure, say so.\n"
        "Context:\n{context}\n\nQuestion: {question}\nAnswer:"
    ),
)

2. Wire up an LLM client

Any object with .generate(prompt: str) -> str works — wrap your provider's SDK if needed:

class OpenAIAdapter:
    def __init__(self, client, model):
        self.client, self.model = client, model
    def generate(self, prompt: str) -> str:
        resp = self.client.chat.completions.create(
            model=self.model, messages=[{"role": "user", "content": prompt}],
        )
        return resp.choices[0].message.content

llm = OpenAIAdapter(openai_client, "gpt-4o-mini")

3. Run the experiment

from zerda.prompt.ab_testing import ABTestRunner

runner = ABTestRunner(llm=llm, variants=[concise, grounded])

exp = runner.run(
    queries=["What is the refund policy?", "How do I reset my password?"],
    contexts=[refund_policy_text, password_reset_text],  # bypass rag_system entirely
    ground_truths=["14 business days", "Use the 'Forgot password' link on the login page."],
    show_progress=True,
)

exp.print_report()

4. Inspect results programmatically

print(exp.winner())
print(exp.all_scores())
for cmp in exp.comparisons:
    print(cmp.query, "->", cmp.best_variant())

5. Promote the winner into PromptEngine

ab_testing.py operates independently of PromptEngine/strategies — it's for comparing arbitrary prompt templates, not zerda.prompt strategy objects directly. To act on the result, translate your finding into a PromptEngine configuration choice (e.g. "the grounded-style variant wins → prefer strategy='cot' with enable_guardrails=True'), since there's no automatic bridge from an ABExperiment winner back into PromptEngine.build() parameters in the current source.

Caveats to know before relying on this in production

• Judge scores are blended with a cheap token-overlap heuristic (0.7 × judge + 0.3 × heuristic) — see api/ab_testing.md#_judge-internal. This dampens (but doesn't eliminate) the effect of judge noise/bias, at the cost of also dampening genuinely correct judge scores that disagree with surface-level word overlap.
• If ground_truths/contexts have fewer entries than queries, extra queries are silently dropped (zip() truncation) rather than erroring.
• rag_system integration expects a specific, non-zerda.prompt.ContextManager-compatible interface — see the warning in api/ab_testing.md. Passing contexts= explicitly sidesteps this entirely, as shown above.