Architecture

On this page

Class hierarchy
Format dispatch flow (AutoLoader)
PDF backend fallback chain
Directory loading sequence
Dependency graph between modules
Execution model: eager vs. lazy
Concurrency model
Extension points
Related pages

Class hierarchy

All loaders ultimately implement the same abstract contract defined in base_loader.py.

AutoLoader and MultimodalAutoLoader are not subclasses of BaseDocumentLoader — they are dispatcher classes whose load() is a classmethod/instance method that constructs and delegates to one of the concrete loaders above.

Format dispatch flow (`AutoLoader`)

PDF backend fallback chain

PDFLoader.load() tries three backends in order and returns the result of the first one that succeeds. This is the only loader in the package with a multi-backend fallback strategy.

Directory loading sequence

Dependency graph between modules

graph_loader and table_loader are self-contained: they only depend on base_loader and the Python standard library (plus lazily-imported third-party parsers), and are not wired into EXTENSION_MAP / AutoLoader's dispatch table — they must be used explicitly.

Execution model: eager vs. lazy

Every loader exposes both load() and lazy_load():

load() always returns a fully-materialized List[LoadedDocument].
lazy_load() is a generator. The default implementation on BaseDocumentLoader just does yield from self.load() (no real streaming benefit), but several loaders override it for true incremental processing:
- CSVLoader.lazy_load() — streams rows via csv.DictReader without loading the whole file into memory as documents up front.
- JSONLinesLoader.lazy_load() — streams line-by-line.
- DirectoryLoader.lazy_load() — yields documents as files complete (sequential mode) or as futures complete (concurrent mode).
- GraphLoader.lazy_load() / TableLoader.lazy_load() — yield nodes/edges or table rows incrementally per parser.

load() in these cases is simply return list(self.lazy_load()).

Concurrency model

Two loaders use thread pools, both via concurrent.futures.ThreadPoolExecutor:

Loader	Concurrency control	Notes
`DirectoryLoader`	`config.max_concurrency` workers (default 4), gated by `config.use_multithreading`	A `threading.Lock` protects the shared `all_docs` list when aggregating results from completed futures.
`MultiURLLoader`	`max_workers` (default 4)	A `threading.Semaphore(1)` plus a shared `last_time` dict enforce a global minimum delay (`delay_between_requests`) between request starts across all worker threads, while the actual HTTP download itself runs outside the semaphore so downloads still overlap.

No loader in this package uses asyncio or multiprocessing; "Async Support" and "Multiprocessing-safe" should be read as not implemented wherever they appear in the API reference.

Extension points

The codebase does not define a formal plugin/registration system (no entry-point discovery, no abstract registry). The two practical extension points are:

Subclass BaseFileLoader or BaseDocumentLoader and implement load(). Your loader is fully compatible with load_and_split() and the rest of the pipeline as long as it returns LoadedDocument objects.
Extend EXTENSION_MAP in config_loader.py and add a branch in DirectoryLoader._get_loader_for_file() / AutoLoader._create_file_loader() if you want a custom loader to participate in directory scanning and auto-detection.