ZerdaZerda

OutputValidator (validator.py)

On this page

Overview

OutputValidator runs a parsed output through up to four independent validation layers and returns a flat list of ValidationResult objects. It never raises on failure — validation failures are reported, not enforced, by this class itself; enforcement (e.g. raising ParseError) is the caller's responsibility (OutputParser._validate only logs failures/warnings, it does not raise).

Depends on schema.py (FieldSchema, ParseTrace, ValidationResult, ValidationStatus) and the standard library (re).

Safety pattern constants

Three module-level compiled-regex constants back Layer 3 (safety checks):

Python · 16 lines
_HALLUCINATION_MARKERS = re.compile(
    r"\b(as of my knowledge cutoff|i cannot verify|i don't actually know|"
    r"i'm making this up|i don't have access to|this is fictional|"
    r"i am hallucinating|fabricated|invented fact)\b", re.IGNORECASE)

_DATA_LEAKAGE_PATTERNS = [
    re.compile(r"\b\d{16}\b"),                                   # credit card
    re.compile(r"\b\d{3}-\d{2}-\d{4}\b"),                        # SSN
    re.compile(r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.\w{2,}\b"), # email
    re.compile(r"(?:password|passwd|secret|api_key)\s*[=:]\s*\S+", re.I),
    re.compile(r"Bearer\s+[A-Za-z0-9\-._~+/]+=*"),               # Bearer token
]

_PROMPT_INJECTION_MARKERS = re.compile(
    r"(ignore (previous|all) instructions|disregard system prompt|"
    r"you are now|pretend you are|act as if|jailbreak|DAN mode)", re.IGNORECASE)

[!NOTE] Pattern coverage is deliberately narrow These are simple keyword/regex heuristics, not an ML-based classifier — they catch obvious, literal phrasing only. A 16-digit number is flagged as a possible credit card regardless of Luhn validity; an SSN-shaped \d{3}-\d{2}-\d{4} pattern is flagged regardless of whether it's actually a Social Security Number. Treat Layer 3 as a first-pass tripwire, not a comprehensive PII/safety scanner.

ValidationRule

Python · 9 lines
class ValidationRule:
    def __init__(
        self,
        name: str,
        predicate: Callable[[Any], bool],
        message: str,
        field: Optional[str] = None,
        severity: ValidationStatus = ValidationStatus.FAILED,
    )

A named, reusable validation predicate.

Methods

  • check(data: Any) -> ValidationResult — calls self.predicate(data). If the predicate itself raises, catches the exception and returns a FAILED result with the exception message embedded (f"Rule '{self.name}' raised exception: {exc}") — your predicate will never propagate an exception up through check(). If the predicate returns falsy, returns a result with status=self.severity (default FAILED, but you can set severity=ValidationStatus.WARNING for soft rules). If truthy, returns PASSED.
Python · 7 lines
rule = ValidationRule(
    "confidence_range",
    lambda d: 0.0 <= float(d.get("confidence", 1.0)) <= 1.0,
    "confidence must be in [0.0, 1.0]",
    field="confidence",
    severity=ValidationStatus.WARNING,
)

OutputValidator

Constructor

Python · 6 lines
OutputValidator(
    fields: Optional[List[FieldSchema]] = None,
    rules: Optional[List[ValidationRule]] = None,
    pydantic_model: Optional[Type] = None,
    enable_safety: bool = True,
)
NameRequiredDefaultNotes
fieldsNo[]Drives Layer 1 (schema completeness/typing).
rulesNo[]Drives Layer 2 (business rules).
pydantic_modelNoNoneDrives Layer 4 (structural Pydantic validation).
enable_safetyNoTrueToggles Layer 3.

Public Methods

validate

Python
def validate(self, data: Any, raw_text: str = "", trace: Optional[ParseTrace] = None) -> List[ValidationResult]

Runs all four layers in order and returns the combined list.

Internal workflow:

  1. Layer 1 — only if self.fields is non-empty AND data is a dict: calls _validate_fields(data).
  2. Layer 2 — for each ValidationRule in self.rules, calls rule.check(data); logs at debug level on failure.
  3. Layer 3 — only if self.enable_safety: computes text_to_check = raw_text or (str(data) if data else "") and calls _safety_checks(text_to_check). Note this means if raw_text is empty, it falls back to str(data) — so safety checks still run against the parsed structure's string representation, not just the original raw LLM text.
  4. Layer 4 — only if self.pydantic_model is set AND data is a dict: calls _pydantic_validation(data).
  5. If a trace is provided, every result (from every layer) is appended via trace.add_validation(r).

Returns: flat List[ValidationResult] — order is Layer 1 results, then Layer 2, then Layer 3, then Layer 4.

Side effects: logger.debug/logger.warning/logger.error/logger.info calls depending on layer and outcome; mutates the trace object if provided (appends to trace.validations).

is_valid

Python
def is_valid(self, data: Any, raw_text: str = "") -> bool

Runs validate() and returns True only if every result's status is PASSED, WARNING, or SKIPPED — i.e. returns False only if at least one result is FAILED. Note: calling this re-runs the entire validation pipeline from scratch (it does not accept a pre-computed results list), so calling both is_valid() and get_failures() on the same data runs validation twice.

get_failures

Python
def get_failures(self, data: Any, raw_text: str = "") -> List[ValidationResult]

Runs validate() and filters to only not r.passed results (i.e. anything that isn't PASSED) — this means WARNING and SKIPPED results are included in get_failures()'s output, since ValidationResult.passed is True only for PASSED. Be careful: despite the name, get_failures() is not restricted to FAILED-status results only.

Private layer methods

  • _validate_fields(data: Dict) -> List[ValidationResult] — for each declared FieldSchema, calls field.validate_value(data.get(field.name)), then (if the value isn't None) calls _check_type. Also emits a WARNING for any key in data not declared in any FieldSchema ("Unexpected field '...' not in schema").
  • _check_type(field_name, value, dtype) -> ValidationResult — maps dtype string to a Python type/tuple ("float" accepts both int and float instances) via isinstance. If the direct isinstance check fails, attempts a soft coercibility check: tries int(value)/float(value)/str(value)/bool(value) and, if that succeeds, returns WARNING (not FAILED) noting the value is coercible. Only truly non-coercible mismatches return FAILED.
  • _safety_checks(text) -> List[ValidationResult] — see safety patterns above. Data leakage checks break after the first pattern match (one FAILED result max for leakage, even if multiple patterns match).
  • _pydantic_validation(data) -> List[ValidationResult] — attempts self.pydantic_model(**data); returns a single PASSED or FAILED result (never both, never a list of per-field pydantic errors — the full pydantic ValidationError message is embedded in the single FAILED result's message).

build_answer_validator

Python
def build_answer_validator(enable_safety: bool = True) -> OutputValidator

Returns a pre-configured OutputValidator for AnswerSchema-shaped data: fields=[answer (required str), sources (optional list), confidence (optional float)], plus two rules — answer_not_empty (FAILED severity) and confidence_range (WARNING severity, checks 0.0 <= confidence <= 1.0).

Python · 3 lines
validator = build_answer_validator()
results = validator.validate({"answer": "Paris", "confidence": 1.4})
# includes a WARNING: "confidence must be in [0.0, 1.0]"

Complexity

O(n) in the size of data's fields for Layers 1/2/4 (bounded by number of declared fields/rules, not input text size); Layer 3 is O(m) in the length of the text being scanned, with a fixed number of regex passes (6 total: 1 hallucination + 4 leakage patterns with early break + 1 injection).

Thread safety

Safe to share a single OutputValidator instance across threads for read-only validate()/is_valid()/get_failures() calls — no instance state is mutated by these methods (the only mutation is to the caller-supplied trace object, not to self).

Best practices

  • Prefer WARNING severity for rules where a soft signal is enough (e.g. "confidence looks off but we can still use the answer") and reserve FAILED for conditions that should actually block usage of the output.
  • Don't rely solely on Layer 3 for PII/security compliance — its patterns are simple regexes and will both under- and over-match; pair with a dedicated PII/secret-scanning tool for production compliance needs.

Common mistakes

  • Calling get_failures() and assuming it only returns FAILED results — it also returns WARNING and SKIPPED. If you specifically want hard failures, filter yourself: [r for r in validator.validate(data) if r.status == ValidationStatus.FAILED].
  • Passing raw_text="" when you have the real raw LLM text available — this silently changes what Layer 3 scans (it falls back to str(data), which won't contain hallucination-admission phrasing that was in the original prose but got stripped out during parsing).

Version History

Not determinable from source.

Related pages