Tutorial: Extending the Library

document_loaders has no formal plugin/registration system — extension happens through ordinary subclassing. This tutorial builds a complete custom loader from scratch and wires it into directory scanning.

Goal

Build a loader for .yaml/.yml files that:

1. Parses the YAML into a dict.
2. Renders it as readable "key: value" text (reusing patterns from JSONLoader).
3. Integrates with DirectoryLoader so .yaml files are picked up during directory scans.

Step 1: Implement the loader

# yaml_loader.py
from typing import List
from zerda.document_loaders.base_loader import BaseFileLoader, LoadedDocument

class YAMLLoader(BaseFileLoader):
    """Load YAML files (.yaml, .yml) as readable text documents."""

    SUPPORTED_EXTENSIONS = [".yaml", ".yml"]

    def __init__(self, file_path: str, encoding: str = "utf-8"):
        super().__init__(file_path=file_path, encoding=encoding)

    def load(self) -> List[LoadedDocument]:
        try:
            import yaml
        except ImportError:
            raise ImportError("PyYAML is required. Install: pip install pyyaml")

        with open(self.file_path, encoding=self.encoding) as f:
            data = yaml.safe_load(f)

        content = self._dict_to_text(data) if isinstance(data, dict) else str(data)

        meta = self._build_file_metadata(
            record_count=len(data) if isinstance(data, dict) else 1,
        )
        return [LoadedDocument(page_content=content, metadata=meta)]

    def _dict_to_text(self, d: dict) -> str:
        return "\n".join(f"{k}: {v}" for k, v in d.items())

This follows the exact conventions observed across every other loader in the package:

• Subclasses BaseFileLoader, giving it free file validation and _build_file_metadata().
• Sets SUPPORTED_EXTENSIONS.
• Lazily imports the third-party dependency (yaml) inside load(), with a clear ImportError message.
• Returns List[LoadedDocument].

Step 2: Test it standalone

with open("config.yaml", "w") as f:
    f.write("name: my-service\nport: 8080\ndebug: true\n")

from yaml_loader import YAMLLoader

docs = YAMLLoader("config.yaml").load()
print(docs[0].page_content)
# "name: my-service\nport: 8080\ndebug: True"
print(docs[0].metadata["record_count"])  # 3

It also gets load_and_split() for free, inherited from BaseDocumentLoader:

chunks = YAMLLoader("config.yaml").load_and_split(text_splitter=my_splitter)

Step 3: Wire it into directory scanning

DirectoryLoader and AutoLoader don't offer a registration hook, so subclass DirectoryLoader to extend both file collection and dispatch:

from zerda.document_loaders import DirectoryLoader
from yaml_loader import YAMLLoader

class ExtendedDirectoryLoader(DirectoryLoader):
    _EXTRA_EXTENSIONS = (".yaml", ".yml")

    def _collect_files(self):
        base_files = super()._collect_files()
        extra_files = [
            f for f in self.directory.rglob("**/*")
            if f.is_file()
            and f.suffix.lower() in self._EXTRA_EXTENSIONS
            and not self._is_excluded(f)
        ]
        return sorted(set(base_files + extra_files))

    def _get_loader_for_file(self, file_path):
        if file_path.suffix.lower() in self._EXTRA_EXTENSIONS:
            return YAMLLoader(str(file_path))
        return super()._get_loader_for_file(file_path)

Now a mixed directory works transparently:

docs = ExtendedDirectoryLoader("./project/").load()
# Picks up .txt, .md, .pdf, ..., AND .yaml/.yml files

Step 4 (optional): add auto-detection support too

If you also want .yaml recognized by something like AutoLoader, note that AutoLoader doesn't support external registration either — you'd write a small wrapper function instead of modifying AutoLoader itself:

from zerda.document_loaders import AutoLoader
from pathlib import Path
from yaml_loader import YAMLLoader

def load_any(source: str, **kwargs):
    if Path(source).suffix.lower() in (".yaml", ".yml"):
        return YAMLLoader(source, **kwargs).load()
    return AutoLoader.load(source, **kwargs)

What you get "for free" by following the base class pattern

By subclassing BaseFileLoader correctly, YAMLLoader automatically has:

• Construction-time validation (FileNotFoundError, extension check)
• load_and_split(text_splitter)
• A default lazy_load() (override it for real streaming if your format benefits — e.g. if you're processing a YAML stream of multiple documents separated by ---)
• Consistent metadata shape (source, loader_type, loaded_at, file_name, file_type, file_size_bytes) matching every other loader's output

Next steps

• Advanced Usage — more extension patterns, including full pipeline construction
• Contributing — conventions to follow if upstreaming a new loader