Core Concepts

The LoadedDocument contract

Every loader in this package — regardless of source format — returns List[LoadedDocument]. This is the single unifying abstraction that lets downstream code (chunkers, embedders, indexers) stay format-agnostic. See LoadedDocument for the full field reference.

LoadedDocument(
    page_content: str,     # the extracted text
    metadata: dict,        # source, loader_type, loaded_at, + format-specific keys
    doc_id: str,            # deterministic, content-derived — stable across runs
)

Eager vs. lazy loading

Every loader exposes both:

• load() — always returns a fully materialized list.
• lazy_load() — a generator. Some loaders (CSVLoader, JSONLinesLoader, DirectoryLoader in sequential mode, GraphLoader, TableLoader) genuinely stream and use less peak memory. Others simply wrap load().

Use lazy_load() whenever you're processing more data than comfortably fits in memory, or when you want to start acting on early results before the whole source finishes loading (sequential DirectoryLoader mode, CSVLoader, JSONLinesLoader).

Configuration objects vs. direct kwargs

Most loaders accept both:

1. A dedicated *Config dataclass via config= — the full option surface.
2. A subset of the most common options as direct constructor kwargs — used to build a default config only if config is None.

# Equivalent:
CSVLoader("data.csv", content_columns=["title"])
CSVLoader("data.csv", config=CSVLoaderConfig(content_columns=["title"]))

Exception to be aware of: GraphLoader and TableLoader have specific keyword-only constructor arguments (include_nodes/include_edges and output_format/per_row respectively) that always override whatever a passed-in config= object specifies for those fields — see the warnings on their respective API pages.

One document per unit of meaning

Loaders generally try to produce one LoadedDocument per the smallest independently meaningful unit of the source:

This granularity is a deliberate design choice for RAG use cases — retrieval works best when each unit is self-contained and semantically coherent, not too large (loses precision) or too small (loses context). Use load_and_split() when you need finer control (e.g. splitting long pages into overlapping chunks).

Auto-detection vs. explicit loaders

• AutoLoader — covers the "core" formats wired into EXTENSION_MAP: text, Markdown, PDF, Word, CSV, Excel, JSON, JSONL, HTML, plus web URLs and directories.
• MultimodalAutoLoader — a parallel dispatcher for OCR/image formats, mixed-content PDFs, and HTML-to-table extraction, falling back to AutoLoader for anything it doesn't specially handle.
• GraphLoader and TableLoader are not covered by either auto-dispatcher — always instantiate them explicitly.

Graceful degradation by design

Several loaders are built around fallback chains rather than hard dependencies:

• PDFLoader tries PyMuPDF → pdfplumber → pypdf, using whichever is installed and succeeds.
• TextLoader/MarkdownLoader try the configured encoding → chardet-detected encoding (if installed) → latin-1 (which never fails).
• DocxLoader on .doc files tries LibreOffice conversion → textract.

This means the package degrades gracefully as dependencies are added or removed, rather than requiring every optional library up front.

Error-handling philosophy

Two consistent patterns appear throughout the codebase:

1. Fail fast at construction, not at load time, for structural problems: BaseFileLoader._validate_file() checks file existence and extension in __init__, so a misconfigured loader raises immediately rather than after other work has already happened.
2. silent_errors/skip_invalid-style flags for content-level problems that are expected to occur in real-world messy data (a malformed CSV encoding, an unparseable JSONL line, one bad file in a large directory) — these default to strict (raise) in most loaders except DirectoryLoader's underlying file iteration, where per-file resilience is more often the desired behavior in batch pipelines. Always check each loader's specific defaults; see Configuration Reference.

Related pages

• Architecture
• Configuration Reference
• API Reference Overview