ZerdaZerda

AutoLoader

On this page

Module: auto_loader · Does not subclass BaseDocumentLoader — a pure dispatcher class with classmethods

Overview

The recommended entry point for most use cases. AutoLoader inspects a source string (file path, URL, or directory path) and routes it to the correct concrete loader, so callers don't need to know or care which loader class handles .pdf vs .csv vs a URL.

Python · 5 lines
from zerda.document_loaders import AutoLoader
docs = AutoLoader.load("report.pdf")
docs = AutoLoader.load("data.csv")
docs = AutoLoader.load("https://example.com/page")
docs = AutoLoader.load("./my_documents/")

Signature

Python · 9 lines
class AutoLoader:
    @classmethod
    def load(cls, source: str, **kwargs) -> List[LoadedDocument]: ...

    @classmethod
    def get_loader(cls, source: str, **kwargs) -> BaseDocumentLoader: ...

    @classmethod
    def detect_type(cls, source: str) -> str: ...

All three are classmethods — AutoLoader is never instantiated; it's used as a namespace, similar to a static utility class.

load(source, **kwargs) -> List[LoadedDocument]

Parameters

NameTypeDescription
sourcestrFile path, URL (http://, https://, ftp://), or directory path
**kwargsanyForwarded verbatim to whichever concrete loader's constructor is selected

Returns

List[LoadedDocument] — equivalent to calling cls.get_loader(source, **kwargs).load().

Raises

  • ValueError — if source is a file with an extension not present in EXTENSION_MAP (message lists all supported extensions)
  • FileNotFoundError — propagated from the underlying loader's construction if a file/directory doesn't exist
  • Any other exception the selected concrete loader's constructor or load() raises

Algorithm — get_loader(source, **kwargs)

Python · 7 lines
source = str(source).strip()
if source.startswith(("http://", "https://", "ftp://")):
    return WebLoader(url=source, **kwargs)
path = Path(source)
if path.is_dir():
    return DirectoryLoader(path=source, **kwargs)
return cls._create_file_loader(path, **kwargs)

Detection order, each check short-circuiting the next:

  1. URL scheme prefix match — startswith(("http://", "https://", "ftp://")). Note: this is a simple prefix check, not a full URL validation; a malformed string that happens to start with "http://" will still be routed to WebLoader and fail at fetch time, not at dispatch time.
  2. Directory check — Path(source).is_dir(). This performs an actual filesystem check, so it requires the path to exist; a non-existent path falls through to file-extension dispatch.
  3. Extension lookup in EXTENSION_MAP, delegating to _create_file_loader, which imports and constructs the matching loader class (TextLoader, MarkdownLoader, PDFLoader, DocxLoader, CSVLoader, ExcelLoader, JSONLoader, JSONLinesLoader, or HTMLLoader) — raising ValueError if the extension is unrecognized.

Examples

Basic — any supported type:

Python · 3 lines
docs = AutoLoader.load("article.md")
docs = AutoLoader.load("dataset.csv")
docs = AutoLoader.load("page.html")

Passing loader-specific options through:

Python · 2 lines
docs = AutoLoader.load("big_report.pdf", per_page=True, password="secret")
# kwargs forwarded to PDFLoader(file_path, per_page=True, password="secret")

URL:

Python
docs = AutoLoader.load("https://docs.python.org/3/")

Directory:

Python · 2 lines
docs = AutoLoader.load("./my_docs/", recursive=False, silent_errors=True)
# kwargs forwarded to DirectoryLoader

Error handling — unsupported extension:

Python · 4 lines
try:
    docs = AutoLoader.load("archive.zip")
except ValueError as e:
    print(e)  # "Unsupported file type: '.zip' for file archive.zip\nSupported extensions: [...]"

Production usage — getting the loader without loading immediately, to configure further:

Python · 4 lines
loader = AutoLoader.get_loader("report.pdf")
print(type(loader).__name__)  # "PDFLoader"
loader.config.per_page = False
docs = loader.load()

get_loader(source, **kwargs) -> BaseDocumentLoader

Same dispatch logic as load(), but returns the constructed, unexecuted loader instance instead of calling .load() on it. Useful when you want to inspect or mutate the loader's config before running it, as shown in the "Production usage" example above.

detect_type(source) -> str

Returns

A short type string, without constructing any loader or touching the loader registry — purely string/path inspection:

  • "web" if the source starts with a recognized URL scheme
  • "directory" if Path(source).is_dir()
  • the LoaderType.value (e.g. "pdf", "csv", "json") if the extension is in EXTENSION_MAP
  • "unknown" otherwise

Examples

Python · 4 lines
AutoLoader.detect_type("report.pdf")        # "pdf"
AutoLoader.detect_type("https://x.com")     # "web"
AutoLoader.detect_type("./data/")           # "directory" (only if it exists and is a dir)
AutoLoader.detect_type("file.xyz")          # "unknown"

This is useful for pre-validating a batch of sources before committing to loading them, e.g. filtering out "unknown" sources up front.

Side effects

None directly — AutoLoader itself performs no I/O. All I/O happens inside whichever concrete loader it delegates to.

Thread safety

Stateless classmethods with no shared mutable state — safe to call concurrently from multiple threads. Thread safety of the delegated loader's load() call depends on that loader's own guarantees (see each loader's page).

Complexity

O(1) dispatch overhead (a few string checks and a dict lookup); total complexity is dominated by whichever concrete loader is selected.

Best practices

  • Prefer AutoLoader.load() for general-purpose ingestion pipelines where the input format varies and you don't want format-specific branching logic scattered through your code.
  • Use AutoLoader.get_loader() instead of .load() when you need to set a config object with non-default values before loading — **kwargs only covers each loader's direct constructor parameters, not nested config object construction.
  • Use detect_type() for fast pre-flight filtering of large source lists before any actual loading begins.

Common mistakes

  • Passing a config= kwarg through AutoLoader.load(source, config=my_config) expecting it to merge with other **kwargs — it works (since **kwargs is forwarded directly to the constructor, and config is a valid named parameter on every concrete loader), but mixing config= with other format-specific kwargs in the same call can be confusing; prefer using get_loader() and configuring the returned instance directly for complex cases.
  • Forgetting that AutoLoader does not know about GraphLoader, TableLoader, or the multimodal_loader classes — these are not wired into EXTENSION_MAP/_create_file_loader and must be imported and used explicitly. See MultimodalAutoLoader for a separate, parallel auto-dispatcher covering OCR/multimodal formats.
  • Relying on detect_type() to validate that a file actually exists — for file extensions it only inspects the suffix string, not the filesystem, so a non-existent .pdf path still returns "pdf".

Related pages