ZerdaZerda

FormatDetector (format_detector.py)

On this page

Overview

FormatDetector guesses the most likely OutputFormat of raw LLM text using a multi-signal scoring system rather than a try/except cascade — every detector method runs independently and contributes a confidence score, so the detector never has to "commit" to a format before evaluating all the evidence.

Depends only on schema.OutputFormat and the standard library (json, re).

FormatCandidate

Python · 5 lines
@dataclass
class FormatCandidate:
    format: OutputFormat
    confidence: float       # 0.0 – 1.0
    evidence: str = ""      # human-readable reason

__repr__: FormatCandidate(json, confidence=0.99, evidence='valid JSON parse succeeded').

FormatDetector

Class attribute

_MIN_CONFIDENCE = 0.15 — candidates scoring below this are discarded by rank().

Public Methods

detect

Python
def detect(self, text: str) -> OutputFormat

Returns just the top-ranked format. Equivalent to self.rank(text)[0].format (or OutputFormat.UNKNOWN if rank() somehow returns nothing — in practice rank() always returns at least one candidate).

Python · 4 lines
detector = FormatDetector()
detector.detect('{"a": 1}')          # OutputFormat.JSON
detector.detect("- item one\n- item two\n- item three")  # OutputFormat.BULLETED_LIST
detector.detect("just a sentence.")  # OutputFormat.PLAIN_TEXT

rank

Python
def rank(self, text: str) -> List[FormatCandidate]

Returns all plausible candidates, deduplicated by format, sorted by confidence descending.

Internal workflow:

  1. Empty/whitespace-only input → returns [FormatCandidate(OutputFormat.UNKNOWN, 0.0, "empty input")] immediately.
  2. Checks for a labeled code fence (```json, ```yaml, etc.) via _detect_from_code_fence. If found with confidence ≥ 0.75, that candidate is placed first and combined with (not replacing) the structural scores from _score_all. If found with lower confidence (unlabeled fence → OutputFormat.UNKNOWN, 0.0), it's merged in but won't outrank real signals.
  3. Runs all ten _score_* methods via _score_all, collecting every non-None result.
  4. Filters to confidence >= _MIN_CONFIDENCE, sorts descending.
  5. Deduplicates by format (first occurrence wins, since the list is sorted by confidence first — so the highest-confidence candidate per format is kept).
  6. If the filtered/deduped list is empty, returns [FormatCandidate(OutputFormat.PLAIN_TEXT, 0.3, "fallback")].
Python · 4 lines
for c in detector.rank(llm_output):
    print(c)
# FormatCandidate(json, confidence=0.99, evidence='valid JSON parse succeeded')
# FormatCandidate(key_value, confidence=0.60, evidence='2 key: value pairs')

detect_with_confidence

Python
def detect_with_confidence(self, text: str) -> Tuple[OutputFormat, float]

Returns (format, confidence) for the top candidate. If rank() returns an empty list (should not happen in practice given the guarantees above), returns (OutputFormat.UNKNOWN, 0.0).

Private scoring methods

Each _score_*(text) -> Optional[FormatCandidate] returns None if the format shows no evidence at all, or a FormatCandidate otherwise. See Concepts → Format detection scoring for the exact heuristics and thresholds of every scorer (_score_json, _score_yaml, _score_csv, _score_markdown_table, _score_numbered_list, _score_bulleted_list, _score_key_value, _score_xml, _score_tool_call, _score_plain_text).

_detect_from_code_fence(text) -> Optional[FormatCandidate] uses _CODE_FENCE_RE = re.compile(r"```(\w*)\s*\n([\s\S]*?)```", re.IGNORECASE) and a hardcoded lang_map (json→0.95, yaml/yml→0.95, csv→0.95, xml→0.90). Unrecognized or empty language tags return a zero-confidence UNKNOWN candidate rather than None, which is a deliberate signal-but-don't-decide design (it still gets merged into the candidate list in rank() but never wins on confidence).

Complexity

Every scorer performs at most a handful of re.findall/re.search passes over the input, each O(n) in text length; rank() therefore runs in O(k·n) where k is the fixed number of scorers (~10). _score_csv's row-consistency check is bounded to the first 4 data rows regardless of total input size, keeping it O(1) beyond header scanning. No recursion, no backtracking-prone regex patterns beyond standard re module behavior.

Thread safety

Fully thread-safe to share a single FormatDetector() instance across threads — it holds no mutable instance state beyond the class-level compiled regex patterns (set once at class definition time) and _MIN_CONFIDENCE.

Best practices

  • If you already know the format with certainty (e.g. you're parsing your own service's structured logs), skip FormatDetector entirely by passing expected_format= to OutputParser — this avoids the (small but nonzero) cost of running all ten scorers.
  • Use rank() instead of detect() when you want to log why a format was chosen, or when you want to try the second-best candidate as a fallback if the top one fails to parse.

Common mistakes

  • Assuming detect() guarantees the text will actually parse successfully as that format — detection is heuristic and confidence-based, not a guarantee. Always be prepared for the subsequent parse step to fail (this is exactly why OutputParser's pipeline has fix/retry stages after detection).
  • OutputFormat — the enum being detected.
  • OutputParser — the primary consumer, via self._detector.detect(text) in _run_pipeline.

Version History

Not determinable from source.

Related pages