RetryHandler (retry_handler.py)

Overview

RetryHandler re-prompts a live LLM with progressively different strategies when parsing keeps failing, until either a parseable response is obtained or the retry budget is exhausted. Depends on the standard library only (json, time, dataclasses, enum) — no dependency on other modules in this package (it receives everything it needs, including the parse function itself, as arguments to run()).

RetryStrategy

class RetryStrategy(Enum):
    STRICT_FORMAT  = auto()
    JSON_STRICT    = auto()
    SIMPLIFIED     = auto()
    GRACEFUL_FAIL  = auto()

See Concepts → Retry strategies for what each prompt-building strategy does.

RetryResult

@dataclass
class RetryResult:
    success: bool
    response: str
    attempts: int = 0
    strategy_used: Optional[RetryStrategy] = None
    errors: List[str] = field(default_factory=list)
    total_duration_ms: float = 0.0

Methods

• as_error_payload(original_query: str = "") -> Dict[str, Any] — returns {"error": True, "message": "Failed to obtain a valid response after all retries", "attempts": ..., "original_query": ..., "errors": [...]}. Useful for returning a structured error to callers/API responses instead of raising.

RetryHandler

Constructor

RetryHandler(
    llm_fn: LLMCallable,
    max_retries: int = 3,
    backoff_seconds: float = 0.5,
    pydantic_schema: Optional[Type] = None,
    required_fields: Optional[List[str]] = None,
)

self._strategies = [STRICT_FORMAT, JSON_STRICT, SIMPLIFIED, GRACEFUL_FAIL] — fixed, not configurable per-instance.

Public Methods

run

def run(
    self,
    original_prompt: str,
    parse_fn: Callable[[str], Any],
    format_instructions: str = "",
    last_error: str = "",
    last_response: str = "",
) -> RetryResult

Attempts up to max_retries LLM calls, testing each response with parse_fn (any callable that raises on failure — this can be a bound method like some_parser.parse, or one of the orchestrator's internal _parse_json/_parse_yaml/etc. functions).

Internal workflow:

1. For attempt_idx from 0 to max_retries - 1:
   • Picks a strategy: self._strategies[min(attempt_idx, len(self._strategies) - 2)] — this indexing means attempts beyond the 3rd (attempt_idx >= 2) keep reusing SIMPLIFIED (index len - 2 = index 2) rather than ever reaching GRACEFUL_FAIL inside the loop; GRACEFUL_FAIL is only ever selected as the strategy for the result returned after the loop exhausts, not chosen mid-loop by this indexing formula (the if strategy == RetryStrategy.GRACEFUL_FAIL: break check exists but is effectively unreachable given the indexing — it's dead code as a loop-exit condition in the current implementation, since min(attempt_idx, 2) never evaluates to 3 (the GRACEFUL_FAIL index) for any attempt_idx >= 0).
   • Builds a retry prompt via _build_retry_prompt(strategy, ...).
   • Sleeps backoff_seconds * attempt_idx (so no sleep on the first attempt, since attempt_idx == 0).
   • Calls self.llm_fn(prompt), increments attempts.
   • Calls parse_fn(raw_response) purely to test parseability — the return value is discarded here (the actual parsed data is not returned by RetryResult; only the raw response string is). If it doesn't raise, returns immediately with success=True and response=raw_response.
   • If parse_fn raises, records the error and continues to the next attempt.
2. If the loop exhausts without success, returns RetryResult(success=False, response="", attempts=attempts, strategy_used=RetryStrategy.GRACEFUL_FAIL, errors=errors, total_duration_ms=...).

Returns: RetryResult. On success, result.response holds the raw LLM text — callers must parse it again themselves (this is what OutputParser._attempt_retry does: data = parse_fn(retry_result.response)).

Raises: Does not raise — all llm_fn/parse_fn exceptions are caught inside the loop and recorded in errors.

Side effects: Makes real calls to self.llm_fn (network/API cost) — up to max_retries calls per run() invocation. Calls time.sleep() between attempts (blocking). Logs at info/warning level per attempt.

handler = RetryHandler(llm_fn=my_llm, max_retries=3)
result = handler.run(
    original_prompt="What is the capital of France?",
    parse_fn=json.loads,
    format_instructions='Return {"answer": "..."}',
    last_error="Expecting value: line 1 column 1",
    last_response="I think it's Paris",
)
if result.success:
    parsed = json.loads(result.response)
else:
    payload = result.as_error_payload("What is the capital of France?")

Private prompt builders

• _build_retry_prompt(strategy, ...) — dispatches to one of the three builders below based on strategy (falls through to original_prompt unchanged for any other value, i.e. effectively for GRACEFUL_FAIL, though as noted above this branch is not reachable via the current loop indexing).
• _strict_format_prompt(original, instructions, error, bad_response) — builds a prompt containing: an apology/retry framing, the previous error (if any), a 300-character preview of the previous bad response (if any), the format instructions (if any), and the original question.
• _json_strict_prompt(original, error) — builds a prompt demanding JSON-only output, embedding a generated schema example (_build_json_example()) and the previous error.
• _simplified_prompt(original) — builds a minimal prompt asking for a concise answer as {"answer": "...", "confidence": 0.9}-shaped JSON, dropping all prior context (previous errors, format instructions) entirely.
• _build_json_example() — if pydantic_schema is set, calls .model_json_schema() and builds a {field: type_placeholder} dict using a hardcoded type_defaults map (string→"example text", integer→0, number→0.0, boolean→True, array→[], object→{}); catches any exception and falls through. Else if required_fields is set, builds {f: f"<{f}_value>" for f in required_fields}. Else returns a hardcoded generic {"answer": "...", "confidence": 0.9} example string.

graceful_fallback

def graceful_fallback(raw_text: str, query: str = "") -> Dict[str, Any]

Module-level function (not a method) — the last-resort fallback used by OutputParser in ParseMode.LENIENT when even retry has failed. Always returns a safe, non-None dict:

{
    "answer": raw_text.strip() if raw_text else "Unable to parse response",
    "sources": [],
    "confidence": 0.0,
    "_parse_error": True,
    "_raw": raw_text[:500] if raw_text else "",
    "_query": query[:200] if query else "",
}

Side effects: logger.error call. Raises: never — this function is designed to be infallible, since it's the final safety net in the pipeline.

Complexity

run() is O(k) LLM calls where k = max_retries, each dominated by your llm_fn's latency (not measured/bounded by this library). Prompt-building and JSON-example generation are O(number of schema fields), negligible relative to LLM call latency.

Thread safety

RetryHandler instances hold only read-only configuration after construction, so sharing one instance across threads for concurrent run() calls is safe as long as your llm_fn is itself thread-safe (e.g. a properly pooled HTTP client). time.sleep() inside run() blocks the calling thread only, not other threads.

Best practices

• Keep backoff_seconds reasonable for interactive use cases (default 0.5s means up to 0.5 + 1.0 = 1.5s of cumulative sleep across 3 retries) — increase it for batch/background jobs where you want to be gentler on rate limits.
• Provide pydantic_schema or required_fields whenever possible — _build_json_example()'s generic fallback ({"answer": "...", "confidence": 0.9}) may not match your actual expected shape, making the JSON_STRICT retry less effective.

Common mistakes

• Assuming a successful RetryResult includes parsed data — it only includes the raw response string (result.response). You must call your parse function on it again.
• Expecting the 4th and later retry attempts to use a GRACEFUL_FAIL-branded prompt — due to the indexing formula in run(), attempts beyond the 2nd index keep reusing the SIMPLIFIED strategy's prompt, not a distinct graceful-fail prompt.

Related APIs

• OutputParser — primary caller, via _attempt_retry.
• ParseTrace — trace.retries is set from retry_result.attempts.

Version History

Not determinable from source.