DirectoryLoader

Module: directory_loader · Extends BaseDocumentLoader directly

Overview

Recursively (or non-recursively) scans a directory, dispatches each supported file to the appropriate format-specific loader based on EXTENSION_MAP, and aggregates the results — optionally in parallel via a thread pool.

Signature

class DirectoryLoader(BaseDocumentLoader):
    def __init__(
        self,
        path: str,
        config: Optional[DirectoryLoaderConfig] = None,
        glob_pattern: str = "**/*",
        recursive: bool = True,
        silent_errors: bool = False,
        show_progress: bool = True,
    ): ...

    def load(self) -> List[LoadedDocument]: ...
    def lazy_load(self) -> Iterator[LoadedDocument]: ...
    def get_stats(self) -> Dict: ...

Parameters

__init__ calls self._validate_directory() immediately, and initializes a threading.Lock() used during concurrent loading.

Returns

List[LoadedDocument] aggregated across all matched, non-excluded, supported files.

Raises

• FileNotFoundError — at construction, if path doesn't exist
• ValueError — at construction, if path exists but is not a directory
• IOError — wrapping any per-file load failure, only if config.silent_errors is False (default behavior re-raises and aborts the whole directory load on the first bad file)

Internal workflow

File collection — _collect_files()

[!TIP] Implementation note The source documents a fix here: the original glob logic was broken for both branches — it stripped the "**/" prefix from the pattern in both the recursive and non-recursive cases, meaning a pattern like "**/*.pdf" never actually matched recursively. The current implementation correctly distinguishes the two cases.

• Recursive (config.recursive=True): strips a leading "**/" from glob_pattern if present, then calls self.directory.rglob(stripped_pattern).
• Non-recursive: takes just the basename portion of the pattern (Path(pattern).name if it contains /) and calls self.directory.glob(base_pattern).
• Filters to files (is_file()) whose lowercased suffix is a key in EXTENSION_MAP, and that are not excluded (_is_excluded).
• Returns a sorted list (deterministic ordering for sequential loading; concurrent loading then processes them in submission order but yields in completion order).

Exclusion — _is_excluded(file_path)

Checks the file's path relative to the scanned directory and its bare filename against every pattern in config.exclude_patterns using fnmatch.fnmatch. A file is excluded if either check matches any pattern.

Loading — sequential vs. concurrent

lazy_load() chooses based on config.use_multithreading and whether there's more than one file:

• Sequential (_load_sequential): iterates files in order, printing a \r-overwriting progress line if show_progress, yielding documents as each file completes.
• Concurrent (_load_concurrent): submits _load_single_file for every file to a ThreadPoolExecutor(max_workers=config.max_concurrency). As futures complete (as_completed), results are appended to a shared list under self._lock, then the whole accumulated list is yielded at the end (not streamed file-by-file in real time, despite being a generator method — yield from all_docs happens only after every future has resolved).

Per-file dispatch — _get_loader_for_file(file_path)

Looks up EXTENSION_MAP[ext] → LoaderType, then constructs the matching concrete loader (TextLoader, MarkdownLoader, PDFLoader, DocxLoader, CSVLoader, ExcelLoader, JSONLoader, JSONLinesLoader, or HTMLLoader) with no extra kwargs — every file is loaded with that loader's defaults. Returns None if the extension has no EXTENSION_MAP entry (filtered out earlier anyway by _collect_files, so this is mostly a defensive fallback).

[!NOTE] No customization per file type Unlike AutoLoader.load(source, **kwargs), DirectoryLoader provides no mechanism to pass custom configuration (e.g. a PDFLoaderConfig with a password) to the loaders it constructs internally. If you need per-format configuration during directory scanning, either subclass DirectoryLoader and override _get_loader_for_file, or scan the directory yourself and call the appropriate loaders directly.

get_stats() -> Dict

Re-runs _collect_files() (does not reuse a cached file list from a prior load() call) and returns:

{
    "total_files": int,
    "by_type": {"pdf": {"count": int, "size_bytes": int}, ...},
    "total_size_bytes": int,
    "total_size_mb": float,  # rounded to 2 decimals
}

Useful for pre-flight inspection of a directory before committing to a full load.

Complexity

• Time: O(files) for collection (glob cost depends on directory size/depth); loading time is the sum of per-file loader costs, parallelized across max_concurrency workers when concurrent mode is active.
• Space: O(total documents across all loaded files) — both load() and the concurrent lazy_load() path materialize the full result list before returning.

Thread safety

DirectoryLoader is internally thread-safe for its own concurrent loading (the threading.Lock() protects the shared results list). It is not designed to have load()/lazy_load() called concurrently from multiple external threads on the same instance.

Side effects

• Prints progress to stdout via print(..., end="") if show_progress=True (both sequential and concurrent modes).
• Logs warnings/errors via the standard logging module for skipped or failed files.

Examples

Basic — recursive scan with defaults:

from zerda.document_loaders import DirectoryLoader

docs = DirectoryLoader("./knowledge_base/").load()

Non-recursive, top-level files only:

docs = DirectoryLoader("./reports/", recursive=False).load()

Resilient batch ingestion — skip bad files instead of aborting:

docs = DirectoryLoader("./mixed_quality_corpus/", silent_errors=True).load()

Custom exclusions and higher concurrency:

from zerda.document_loaders import DirectoryLoader, DirectoryLoaderConfig

config = DirectoryLoaderConfig(
    exclude_patterns=["*.tmp", "drafts/*", "__pycache__/*"],
    max_concurrency=8,
)
docs = DirectoryLoader("./project_docs/", config=config).load()

Pre-flight inspection before a full load:

loader = DirectoryLoader("./large_archive/")
stats = loader.get_stats()
print(f"{stats['total_files']} files, {stats['total_size_mb']} MB")
print(stats["by_type"])
# Only call loader.load() after confirming this looks reasonable

Error handling — strict mode (default):

try:
    docs = DirectoryLoader("./corpus/").load()
except IOError as e:
    print(f"Aborted on first bad file: {e}")

Best practices

• Use silent_errors=True for large, heterogeneous, untrusted corpora where a single malformed file shouldn't block the whole batch — but log and review the warnings afterward, since silently-skipped files mean silently-missing data.
• Check get_stats() before loading very large directories to estimate scope and catch unexpected file-type distributions early.
• Tune max_concurrency to your I/O profile — CPU-bound parsing (PDF/DOCX) benefits less from high thread counts than I/O-bound formats, and very high concurrency can also strain backends like LibreOffice subprocess conversion (used by DocxLoader for .doc files).

Common mistakes

• Expecting per-loader configuration (passwords, custom delimiters, column selection) to apply during directory scanning — it doesn't; every file is loaded with default settings for its loader class.
• Assuming lazy_load() in concurrent mode streams results progressively — it actually waits for all futures to complete before yielding anything, since yield from all_docs happens after the with ThreadPoolExecutor(...) block exits. Use sequential mode if true streaming/early-yield behavior matters to your application.
• Forgetting that unsupported extensions are silently skipped (not even logged) — _collect_files() filters them out before any per-file processing occurs.

Related APIs

• DirectoryLoaderConfig
• EXTENSION_MAP
• AutoLoader — dispatches to DirectoryLoader automatically when given a directory path
• LoadedDocument

Version History

Cannot be determined from the repository, beyond the documented fix to the recursive/non-recursive glob logic noted in _collect_files()'s docstring.