ZerdaZerda

Integrations

On this page

document_loaders produces plain LoadedDocument objects with no hard dependency on any specific downstream framework — it's designed to sit at the front of a pipeline, feeding whatever chunking, embedding, and vector-store tooling you use next.

Chunking / text splitters

BaseDocumentLoader.load_and_split(text_splitter=None) accepts any object exposing a split_text(text: str) -> List[str] method — this is a duck-typed interface, not a specific class from this package or any particular third-party library.

Python · 11 lines
from zerda.document_loaders import PDFLoader

class SimpleSplitter:
    def __init__(self, chunk_size=500):
        self.chunk_size = chunk_size

    def split_text(self, text: str):
        return [text[i:i + self.chunk_size] for i in range(0, len(text), self.chunk_size)]

loader = PDFLoader("report.pdf")
chunks = loader.load_and_split(text_splitter=SimpleSplitter(chunk_size=500))

Any chunker from the broader Fennec ecosystem, LangChain, LlamaIndex, or a hand-rolled splitter works as long as it implements split_text.

Vector stores / embedding pipelines

LoadedDocument.to_dict() produces a plain {"doc_id", "page_content", "metadata"} dict — a natural fit for most vector-store client APIs that expect id/text/metadata-shaped records:

Python · 11 lines
docs = AutoLoader.load("knowledge_base.pdf")
records = [doc.to_dict() for doc in docs]

# Example: adapting to a generic vector store client
for record in records:
    vector_store.upsert(
        id=record["doc_id"],
        text=record["page_content"],
        metadata=record["metadata"],
        vector=embed(record["page_content"]),
    )

Because doc_id is content-derived and deterministic (see LoadedDocument), re-running ingestion on unchanged source files produces the same IDs — enabling idempotent upserts rather than duplicate insertion.

GraphRAG pipelines

GraphLoader and InMemoryGraphLoader are purpose-built to feed graph-aware RAG pipelines, where nodes and edges need to be independently retrievable and embeddable, while metadata (element_type, node_id, source_node, target_node, relation) preserves the graph structure for downstream graph-traversal or hybrid vector+graph retrieval strategies.

Zerda ecosystem

multimodal_loader imports its base classes via zerda.document_loaders.base_loader, indicating document_loaders ships as a submodule within the broader zerda package alongside other RAG-pipeline components (chunking, embeddings, orchestration). See Installation for import-path notes.

What this package does not do

To set expectations clearly:

  • No built-in embedding — document_loaders produces text and metadata only; embedding is entirely out of scope.
  • No built-in vector storage — nothing here writes to a vector database.
  • No built-in chunking algorithm — load_and_split() is a thin pass-through to a caller-supplied splitter; there is no bundled text-splitting implementation in this module.
  • No LLM calls — no summarization, extraction, or generation happens inside this package (OCR is the closest thing to "AI" in this module, and it's a fixed Tesseract pipeline, not an LLM).

Related pages