Architecture

On this page

Package layout
The contract: BaseLLMInterface
prompt vs messages — one signature, seven providers
Configuration flow
Client lifecycle
Related pages

Package layout

There is no plugin system, no dependency-injection container, no registry that dynamically looks up a provider by string — __llm_providers__ in __init__ is a plain list of provider name strings (["openai", "anthropic", "ollama", "gemini", "mistral", "groq"]) for reference only; nothing in the package reads it to instantiate a class. Selecting a provider means importing its class directly.

llm/tools.py is a standalone, provider-agnostic module with no dependency on BaseLLMInterface or any SDK — it exports the ToolCall dataclass, one _normalize_tool() helper, seven to_*_tools() schema converters (one per provider, with to_groq_tools/to_mistral_tools/to_ollama_tools being plain aliases of to_openai_tools since those three providers share OpenAI's function-calling dialect), five parse_*_tool_calls() response parsers, and tool_calls_to_dicts(). Every provider module imports only the specific converter/parser pair it needs from this module. See Tool Calling for the full contract.

The contract: `BaseLLMInterface`

Every provider class subclasses BaseLLMInterface, an ABC defined in base_llm_interface.py. It fixes:

Constructor shape: __init__(self, model_name: str, api_key: str, **kwargs) — subclasses call super().__init__(...) then do their own SDK client setup.
Two abstract methods that every subclass must implement: generate(...) (sync) and generate_async(...) (async). Both are declared with tools: Optional[list] = None and tool_choice: Optional[Union[str, dict]] = None parameters — every one of the seven subclasses implements this same signature, converting tools via the matching llm.tools.to_*_tools() helper and parsing the response via the matching parse_*_tool_calls() helper. See Tool Calling.
One concrete method with a default implementation: astream(...), an async generator. The base implementation calls generate_async() for the full response and then yields it word-by-word with await asyncio.sleep(0) between words — a fallback, not real token streaming. Providers that support real streaming (OpenAI, Anthropic, Groq, AWS Bedrock) override astream entirely.
Async context manager support: __aenter__ / __aexit__, where __aexit__ calls acleanup(). The base acleanup() is a no-op pass — subclasses override it to close SDK client sessions.
validate_connection(...): a concrete helper that fires a small test prompt (sync or async) and returns a dict (success, reason, response) instead of letting exceptions propagate — useful for health checks.

`prompt` vs `messages` — one signature, seven providers

BaseLLMInterface.generate (and generate_async, astream) is declared as generate(self, messages: Optional[list] = None, prompt: Optional[str] = None, max_tokens=..., temperature=..., **kwargs), and every one of the seven provider subclasses implements this same signature — messages is accepted as the first parameter, ahead of prompt, everywhere. Internally each provider resolves the pair through its own _build_messages (or, for Gemini, _build_contents) static helper: messages wins if both are given; if neither is given, the helper raises ValueError("Either 'prompt' or 'messages' must be provided.").

What still differs by provider is (a) the exact payload shape sent to the underlying SDK/HTTP call, and (b) whether that ValueError reaches the caller as an exception or gets folded into the provider's normal error return — see the note below.

Payload shape per provider:

OpenAI, Anthropic, Groq, Mistral — pass the messages list straight through, unmodified: flat OpenAI-style {"role": ..., "content": "..."} dicts.
AWS Bedrock — prompt is auto-wrapped into Bedrock's Converse content-block shape ({"role": "user", "content": [{"text": prompt}]}); a messages list you supply yourself must already be in that block shape.
Gemini — _build_contents translates OpenAI-style messages into Gemini's shape: assistant → model (Gemini has no assistant role), and any system-role entries are pulled out of the turn history and returned separately as system_instruction rather than left in contents.
Ollama — generate/generate_async/astream post to {base_url}/api/chat with the messages list when messages is given, and to {base_url}/api/generate with prompt otherwise — it's the only provider that changes endpoint, not just payload shape, based on which argument you pass.

[!NOTE] Whether the missing-prompt/messages ValueError propagates depends on the call path On most generate/generate_async/astream methods, the _build_messages/_build_contents call sits inside that method's own try/except, so a missing prompt/messages is caught there along with every other exception and turned into the provider's normal error return (a dict for OpenAI/Anthropic/Groq/AWS Bedrock; an error-prefixed str for Mistral, Gemini's async path, and Ollama's astream when aiohttp is installed) — it does not reach the caller as a raised exception. The exceptions to that are GeminiInterface.generate (sync) and OllamaInterface.generate/generate_async, which build the messages list before entering their retry/network try/except, so on those two providers' sync-and-async generate methods the ValueError really does propagate uncaught. See Troubleshooting for the full per-method breakdown.

Practically: always call with keyword arguments (llm.generate(prompt=...) or llm.generate(messages=...)) rather than positionally, since messages is the first positional parameter on every provider — passing a bare string positionally would silently be treated as messages rather than prompt.

Configuration flow

Each interface module creates its own llm_config() instance at import time (some name it config, others llm_config_ — see Configuration) and uses its fields as default parameter values in method signatures. Because Python evaluates default argument values once, at function-definition time, these defaults are fixed when the module is imported — mutating an llm_config instance's fields afterward will not retroactively change already-defined method defaults. See Configuration for the practical implication.

Client lifecycle

Most providers (OpenAI, Anthropic, Groq) create both a sync and an async SDK client eagerly in __init__. AWS Bedrock creates the sync boto3 client eagerly but the async aioboto3 client lazily, on first generate_async/astream call (_get_async_client), since aioboto3 is an optional dependency. Ollama does not use an SDK client at all — it talks to a local HTTP server via requests/aiohttp and can manage that server's process lifecycle itself (see OllamaInterface).

Cleanup is not automatic except for Ollama (which stops a self-started server in __del__). For the other providers, call cleanup() (sync) / await acleanup() (async), or use the async with context manager.