ZerdaZerda

Contributing

On this page

Observed coding conventions

Based on static analysis of all fourteen modules in document_loaders/:

  • One class per format, extending BaseFileLoader (file-backed) or BaseDocumentLoader (everything else).
  • Config dataclasses for anything with more than 2–3 options, named <Format>LoaderConfig, defined either in config_loader.py (for the "core" formats) or alongside the loader itself (GraphLoaderConfig in graph_loader.py, TableLoaderConfig in table_loader.py).
  • Lazy imports of third-party dependencies inside methods (not at module top-level), each wrapped in a try/except ImportError that raises a clear, actionable message naming the exact pip install command. This is consistent across every optional-dependency loader in the package.
  • SUPPORTED_EXTENSIONS as a class attribute on every file-based loader, used by BaseFileLoader._validate_file() for eager validation.
  • _build_file_metadata(**extra) (or _build_metadata for non-file loaders) as the standard way to construct metadata, ensuring every loader's output includes the common base fields (source, loader_type, loaded_at, plus file-specific fields where applicable).
  • Bilingual comments appear in graph_loader.py and table_loader.py (Arabic-first docstrings/comments); other modules use English-only or English-with-Arabic-translation-in-parentheses docstrings. There's no single enforced convention across the whole codebase — follow the style of whichever file you're modifying.
  • # FIX: comments document specific bug fixes inline, explaining what was wrong before and why the current approach is correct — a useful pattern to continue for any bug fix, since this repository has no separate changelog (see Changelog).

Adding a new loader

Based on the pattern every existing loader follows:

  1. Create a new module (or add to an existing one if closely related to it).
  2. Define a <Format>LoaderConfig dataclass if the loader has more than a couple of options.
  3. Subclass BaseFileLoader (set SUPPORTED_EXTENSIONS) or BaseDocumentLoader directly.
  4. Implement load(), returning List[LoadedDocument]. Use self._build_file_metadata(**extra) / self._build_metadata(source=..., **extra) for metadata.
  5. Lazily import any third-party dependency inside load() (or a helper method), wrapped in try/except ImportError with a clear install message.
  6. If the format benefits from true streaming, override lazy_load() as a generator and have load() call return list(self.lazy_load()).
  7. Add the new class to document_loaders/__init__.py's imports and __all__.
  8. If the format should be auto-detected, add an entry to EXTENSION_MAP in config_loader.py and a corresponding branch in AutoLoader._create_file_loader() and DirectoryLoader._get_loader_for_file().
  9. Add a page to this documentation project under docs/api/ following the structure of the existing pages, and link it from docs/api/overview.md and mkdocs.yml's navigation.

Testing

No test suite (tests/, pytest.ini, etc.) was present in the analyzed source tree, so no testing conventions can be documented from the repository itself. If adding tests, mirror the module structure (one test module per loader module) as a reasonable default, and ensure coverage of:

  • The happy path for each supported format/variant.
  • Missing-dependency ImportError behavior (can be tested by temporarily hiding the import, e.g. via sys.modules manipulation in a fixture).
  • Malformed-input handling (invalid JSON, corrupted PDF, wrong password, unsupported extension).
  • Metadata field presence and correctness.
  • For loaders with a documented # FIX: history (see Changelog), a regression test guarding against reintroducing the specific bug described.

Related pages