ZerdaZerda

Performance

On this page

This page summarizes performance-relevant behavior that can be determined directly from the implementation. No benchmark numbers are provided, since none exist in the source repository — all statements here are about algorithmic/structural characteristics, not measured throughput.

Memory: streaming vs. full materialization

Loaderlazy_load() is true streaming?Notes
TextLoader, MarkdownLoaderNoWhole file read into memory as one string regardless
PDFLoaderNoWhole requested page range extracted into memory before returning
DocxLoaderNoWhole document parsed and joined into memory
CSVLoaderYescsv.DictReader streams rows; lazy_load() yields one document at a time without holding the full file in memory
ExcelLoaderNo_load_sheet/_load_with_pandas accumulate all rows for a sheet before returning, though openpyxl's read_only=True mode itself streams from disk internally
JSONLoaderNoWhole file parsed via json.load() up front
JSONLinesLoaderYesReads and yields line-by-line; well-suited to very large files
HTMLLoader, HTMLStringLoaderNoWhole document parsed into a BeautifulSoup tree in memory
WebLoaderNoWhole response body held in memory
MultiURLLoaderNoAll results accumulated before load() returns; no lazy variant
DirectoryLoaderPartiallySequential mode yields as each file completes; concurrent mode waits for all futures before yielding anything (see Directory loader caveat)
GraphLoaderPartiallyThe underlying file (JSON/XML) is fully parsed into memory first, but lazy_load() yields node/edge documents incrementally after that
TableLoaderPartiallyEach individual table is built fully in memory (_RawTable), but tables are yielded as each one is processed rather than waiting for the whole file

Takeaway: for very large inputs, prefer CSVLoader and JSONLinesLoader (or the equivalent input formats) where true streaming matters, and use DirectoryLoader in sequential mode (use_multithreading=False) rather than concurrent mode if you need progressive yielding across many files.

Concurrency

Two loaders use ThreadPoolExecutor:

  • DirectoryLoader — max_concurrency workers (default 4). Best suited to I/O-bound or mixed workloads; CPU-heavy parsing (large PDFs via PyMuPDF, .doc conversion via LibreOffice subprocess) may not scale linearly with worker count due to the GIL for pure-Python parsing paths, though PyMuPDF and similar C-extension libraries release the GIL during their native calls.
  • MultiURLLoader — max_workers (default 4), plus a global semaphore-enforced minimum delay between request starts. This is genuinely effective for I/O-bound network fetches, where threads spend most of their time waiting on the network.

No asyncio or multiprocessing is used anywhere in this package — see Architecture → Concurrency model.

PDF backend selection affects both speed and quality

PDFLoader tries PyMuPDF first, then pdfplumber, then pypdf. Per the module docstring, PyMuPDF is described as the fastest backend with the best Arabic/RTL extraction quality. If speed matters and multiple backends are installed, PyMuPDF being available and succeeding is the best-case path — there's no way to force backend selection except by controlling which packages are installed in the environment (there's no backend= parameter).

OCR is comparatively expensive

ImageOCRLoader and PDFMultimodalLoader (with ocr_images=True) invoke Tesseract per image, which is orders of magnitude slower than digital text extraction. For PDFs with a mix of scanned and digital pages:

  • Setting ocr_images=False in PDFMultimodalLoader skips OCR entirely, behaving similarly in cost to plain PDFLoader.
  • Raising min_image_size in PDFMultimodalLoader skips OCR on small decorative images (logos, icons), reducing wasted OCR calls.
  • preprocessing=True in ImageOCRLoader adds a grayscale + threshold pass before OCR — a comparatively cheap operation relative to the OCR call itself.

DirectoryLoader.get_stats() for pre-flight sizing

Before committing to a full directory load, DirectoryLoader.get_stats() re-runs file collection and reports file counts and total size by type — useful for estimating scope and catching unexpectedly large or oddly-distributed corpora before spending time (and OCR/API costs, if applicable) loading everything.

Python · 4 lines
loader = DirectoryLoader("./huge_archive/")
stats = loader.get_stats()
if stats["total_size_mb"] > 5000:
    print("Consider batching or filtering before a full load")

Note this method re-scans the filesystem (_collect_files() runs again) rather than reusing a previous scan — calling it repeatedly has the same glob cost as load()'s own initial file collection.

doc_id determinism enables cheap incremental ingestion

Because LoadedDocument.doc_id is a deterministic hash of (source, page_number, page_content) (see LoadedDocument), re-running ingestion over an unchanged corpus produces identical IDs. This lets a pipeline skip re-embedding/re-indexing documents whose ID already exists in the target store, without needing separate change-detection logic (as long as the vector store/index tracks which doc_ids it has already processed).

Where this package does not optimize for you

  • No built-in caching of loaded documents across calls — every load() re-reads and re-parses the source from scratch.
  • No built-in rate limiting for anything except MultiURLLoader's HTTP fetches — hitting an external API-backed loader repeatedly is entirely the caller's responsibility to throttle.
  • No batching/pagination controls beyond what individual formats naturally offer (e.g. PDF start_page/end_page, CSV max_rows) — there's no universal "load N documents at a time across an arbitrary source" primitive.

Related pages