ZerdaZerda

LoadedDocument, BaseDocumentLoader, BaseFileLoader

On this page

Module: base_loader

These three types form the foundation every other loader in the package builds on.

LoadedDocument

Python · 5 lines
@dataclass
class LoadedDocument:
    page_content: str
    metadata: Dict[str, Any] = field(default_factory=dict)
    doc_id: Optional[str] = None

Overview

The universal unit of output for every loader in this package. Conceptually equivalent to a "Document" type in other RAG/LLM frameworks (e.g. LangChain's Document).

Fields

FieldTypeDefaultDescription
page_contentstrrequiredThe extracted text
metadataDict[str, Any]{}Source path, loader type, timestamps, and format-specific fields (page number, row number, sheet name, etc.)
doc_idOptional[str]NoneAuto-generated if not supplied (see below)

__post_init__ — deterministic ID generation

Python · 6 lines
def __post_init__(self):
    if not self.doc_id:
        source = self.metadata.get("source", "")
        page = str(self.metadata.get("page_number", ""))
        seed = f"{source}:{page}:{self.page_content}"
        self.doc_id = "doc_" + hashlib.sha256(seed.encode()).hexdigest()[:16]

Algorithm: if no doc_id is supplied at construction, one is derived by SHA-256 hashing "{source}:{page}:{content}" and truncating to the first 16 hex characters, prefixed with doc_.

[!TIP] Implementation note A docstring comment in the source explicitly documents that this replaced an earlier version which used time.time() to generate IDs — meaning identical documents got a different ID on every run, breaking deduplication and caching. The current scheme is purely content-based and therefore stable across runs: loading the same file twice produces identical doc_id values for identical pages, which is useful for incremental indexing (skip-if-exists) and deduplication.

Caveat: the ID only incorporates source and page_number from metadata, not the full metadata dict. Two documents with identical content and source but different metadata (e.g. different row_number) will collide if page_number also matches or is absent in both — doc_id uniqueness is not guaranteed across all metadata variations, only across (source, page_number, page_content) triples.

Methods

to_dict() -> Dict[str, Any]

Returns {"doc_id": ..., "page_content": ..., "metadata": ...}. Useful for JSON serialization or passing to systems expecting plain dicts.

__len__() -> int

Returns len(self.page_content) — character count, not token count.

__eq__(other) -> bool

Two LoadedDocument instances are equal iff their doc_id matches — content and metadata are not compared directly (though they factor into doc_id indirectly).

__hash__() -> int

hash(self.doc_id) — makes LoadedDocument usable in sets and as dict keys, consistent with __eq__.

__repr__() -> str

Returns a readable preview: LoadedDocument(doc_id='doc_...', chars=N, preview='first 80 chars...').

Example

Python · 9 lines
from zerda.document_loaders.base_loader import LoadedDocument

doc = LoadedDocument(
    page_content="Cairo is the capital of Egypt.",
    metadata={"source": "facts.txt", "page_number": 1},
)
print(doc.doc_id)        # doc_<16 hex chars>, stable across runs
print(len(doc))          # 31
print(doc.to_dict())

BaseDocumentLoader (abstract)

Python · 5 lines
class BaseDocumentLoader(ABC):
    def __init__(self, encoding: str = "utf-8", **kwargs): ...

    @abstractmethod
    def load(self) -> List[LoadedDocument]: ...

Overview

The root abstract base for every loader in the package, including non-file-based ones (WebLoader, DirectoryLoader, HTMLStringLoader, InMemoryGraphLoader, InMemoryTableLoader). Defines the contract: a loader is anything with a load() method returning List[LoadedDocument].

Constructor

ParameterTypeDefaultDescription
encodingstr"utf-8"Stored on self.encoding; used by subclasses that read text
**kwargsany—Captured into self._extra_config (not otherwise consumed in this base class)

Abstract method

load() -> List[LoadedDocument]

Must be implemented by every subclass. This is where format-specific extraction logic lives.

Concrete methods

lazy_load() -> Iterator[LoadedDocument]

Default implementation: yield from self.load(). Subclasses that benefit from true incremental processing (CSVLoader, JSONLinesLoader, DirectoryLoader, GraphLoader, TableLoader) override this for real streaming; everything else gets eager-load-then-iterate behavior with no memory savings.

load_and_split(text_splitter=None) -> List[LoadedDocument]

Python · 12 lines
def load_and_split(self, text_splitter=None) -> List[LoadedDocument]:
    documents = self.load()
    if text_splitter is None:
        return documents
    chunked = []
    for doc in documents:
        chunks = text_splitter.split_text(doc.page_content)
        for i, chunk in enumerate(chunks):
            meta = doc.metadata.copy()
            meta.update({"chunk_index": i, "total_chunks": len(chunks), "original_doc_id": doc.doc_id})
            chunked.append(LoadedDocument(page_content=chunk, metadata=meta))
    return chunked

Algorithm: loads all documents, then for each one calls text_splitter.split_text(doc.page_content) — text_splitter is a duck-typed object, not a specific class from this package; any object with a split_text(str) -> List[str] method works. Each resulting chunk becomes a new LoadedDocument with metadata copied from the parent plus chunk_index, total_chunks, and original_doc_id.

Raises: AttributeError if text_splitter does not implement split_text. Whatever self.load() can raise also propagates.

_build_metadata(source: str, **extra) -> Dict[str, Any] (protected)

Builds the common metadata envelope: {"source": ..., "loader_type": self.__class__.__name__, "loaded_at": "YYYY-MM-DD HH:MM:SS"}, merged with extra. Used by non-file loaders (web, directory, graph/table in-memory variants).

Example: implementing a custom loader

Python · 13 lines
from zerda.document_loaders.base_loader import BaseDocumentLoader, LoadedDocument

class StaticStringLoader(BaseDocumentLoader):
    def __init__(self, text: str, source: str = "manual"):
        super().__init__()
        self.text = text
        self.source = source

    def load(self):
        meta = self._build_metadata(source=self.source)
        return [LoadedDocument(page_content=self.text, metadata=meta)]

docs = StaticStringLoader("Hello world").load()

BaseFileLoader(BaseDocumentLoader) (abstract)

Python · 4 lines
class BaseFileLoader(BaseDocumentLoader):
    SUPPORTED_EXTENSIONS: List[str] = []

    def __init__(self, file_path: str, encoding: str = "utf-8", **kwargs): ...

Overview

Specializes BaseDocumentLoader for loaders backed by a single file path. Adds path validation and file-specific metadata building. Every concrete file-based loader in the package (TextLoader, PDFLoader, DocxLoader, CSVLoader, ExcelLoader, JSONLoader, JSONLinesLoader, HTMLLoader, GraphLoader, TableLoader, and the multimodal loaders) subclasses this.

Class attribute

SUPPORTED_EXTENSIONS: List[str] — overridden by each subclass; an empty list (the base default) disables extension checking entirely.

Constructor

ParameterTypeDefaultDescription
file_pathstrrequiredPath to the file; wrapped in pathlib.Path
encodingstr"utf-8"Passed up to BaseDocumentLoader
**kwargsany—Forwarded to BaseDocumentLoader

Calls self._validate_file() automatically during __init__.

_validate_file() (protected)

Raises:

  • FileNotFoundError — if file_path does not exist
  • ValueError — if the path exists but is not a regular file (e.g. it's a directory)
  • ValueError — if SUPPORTED_EXTENSIONS is non-empty and the file's suffix (lowercased) is not in that list

This validation runs eagerly at construction time, not at load() time — instantiating a loader for a missing file fails immediately.

_build_file_metadata(**extra) -> Dict[str, Any] (protected)

Python · 10 lines
def _build_file_metadata(self, **extra) -> Dict[str, Any]:
    stat = self.file_path.stat()
    meta = self._build_metadata(
        source=str(self.file_path.resolve()),
        file_name=self.file_path.name,
        file_type=self.file_path.suffix.lower(),
        file_size_bytes=stat.st_size,
    )
    meta.update(extra)
    return meta

Every file-based loader's load() method calls this to populate the base metadata fields (source, loader_type, loaded_at, file_name, file_type, file_size_bytes) before adding its own format-specific keys.

Example

Python · 9 lines
from zerda.document_loaders.base_loader import BaseFileLoader, LoadedDocument

class FirstLineLoader(BaseFileLoader):
    SUPPORTED_EXTENSIONS = [".txt"]

    def load(self):
        first_line = self.file_path.read_text(encoding=self.encoding).splitlines()[0]
        meta = self._build_file_metadata(note="first line only")
        return [LoadedDocument(page_content=first_line, metadata=meta)]

Thread safety & async support

Neither LoadedDocument, BaseDocumentLoader, nor BaseFileLoader perform any locking or async I/O. LoadedDocument is an effectively-immutable-by-convention dataclass (nothing prevents mutation, but nothing in the package mutates an instance after construction except load_and_split's copy-then-build pattern, which never mutates an existing instance). Concrete loaders that need concurrency (DirectoryLoader, MultiURLLoader) implement their own locking — see Architecture → Concurrency model.

Version History

Cannot be determined from the repository — no changelog or git history was available for analysis. The LoadedDocument.__post_init__ docstring documents one specific historical fix (switching from time-based to content-based ID generation) but gives no version or date.

Related pages