ZerdaZerda

Configuration Reference

On this page

All configuration in document_loaders is expressed as dataclasses in config_loader.py. Every loader accepts a config object or a subset of common fields as direct constructor keyword arguments (which the loader uses to build a default config internally). Passing config= explicitly always takes precedence — direct kwargs are only used when config is None.

LoaderType (Enum)

Identifies every loader format the package understands for dispatch purposes (used by EXTENSION_MAP, AutoLoader, and DirectoryLoader).

Python · 14 lines
class LoaderType(Enum):
    TEXT = "text"
    PDF = "pdf"
    DOCX = "docx"
    CSV = "csv"
    EXCEL = "excel"
    JSON = "json"
    JSONL = "jsonl"
    HTML = "html"
    MARKDOWN = "markdown"
    WEB = "web"
    SITEMAP = "sitemap"
    DIRECTORY = "directory"
    AUTO = "auto"

[!WARNING] Declared but unused values SITEMAP and AUTO are defined in the enum but no loader implementation maps to them in EXTENSION_MAP or AutoLoader._create_file_loader(). There is no sitemap-parsing loader in this codebase despite the enum value existing.

EXTENSION_MAP

Dict[str, LoaderType] — the single source of truth for file-extension → loader-type dispatch, used by both AutoLoader and DirectoryLoader.

ExtensionLoader Type
.txtTEXT
.md, .markdownMARKDOWN
.pdfPDF
.docx, .docDOCX
.csv, .tsvCSV
.xlsx, .xlsEXCEL
.jsonJSON
.jsonl, .ndjsonJSONL
.html, .htmHTML

TextLoaderConfig

Used by TextLoader.

FieldTypeDefaultPurpose
encodingstr"utf-8"Preferred file encoding
autodetect_encodingboolTrueFall back to chardet detection, then latin-1, if the preferred encoding fails to decode
errorsstr"replace"Passed to str.decode/Path.read_text — "strict", "ignore", or "replace"

PDFLoaderConfig

Used by PDFLoader.

FieldTypeDefaultPurpose
extract_imagesboolFalseDeclared but not read anywhere in pdf_loader.py — has no effect in the current implementation. Use PDFMultimodalLoader for actual image OCR.
passwordOptional[str]NonePassword for encrypted PDFs
page_separatorstr"\n\n"Joiner between pages when per_page=False
per_pageboolTrueOne LoadedDocument per page vs. one document for the whole file
start_pageint0Zero-indexed first page to include
end_pageOptional[int]NoneExclusive end page index; None = last page

DocxLoaderConfig

Used by DocxLoader.

FieldTypeDefaultPurpose
include_tablesboolTrueAppend table content (rendered as text) after paragraphs
table_separatorstr"\n"Joiner between table rows
row_separatorstr" | "Joiner between cells within a row
include_headersboolTruePrepend [Header]: ... text extracted from section headers
include_footersboolTrueAppend [Footer]: ... text extracted from section footers

CSVLoaderConfig

Used by CSVLoader.

FieldTypeDefaultPurpose
delimiterstr","Field delimiter (auto-switched to "\t" for .tsv files unless explicitly overridden)
quote_charstr'"'CSV quote character
encodingstr"utf-8"File encoding (with a single latin-1 fallback retry on decode failure)
source_columnOptional[str]NoneColumn whose value overrides metadata["source"] per row
content_columnsOptional[List[str]]NoneColumns to include in page_content; None = use all columns
metadata_columnsOptional[List[str]]NoneColumns to copy into metadata instead of content
skip_rowsint0Rows to skip before the header (consumed via next(f, None), so this skips raw lines, not parsed rows)
max_rowsOptional[int]NoneMaximum data rows to read
row_joinerstr"\n"Joiner between "column: value" fragments in page_content

JSONLoaderConfig

Used by JSONLoader. (JSONLinesLoader does not use this dataclass — it takes its equivalent options as direct constructor parameters.)

FieldTypeDefaultPurpose
jq_schemaOptional[str]NoneA jq-style path expression (uses the jq package if installed, else a simplified built-in dot-path parser) applied to the parsed JSON before record extraction
content_keyOptional[str]NoneDict key to use as page_content; falls back to stringifying the whole record
metadata_funcOptional[Callable[[Dict], Dict]]NoneCallable that receives the raw record dict and returns extra metadata fields
text_contentboolTrueIf False (and no content_key match), the record is serialized as pretty-printed JSON instead of "key: value" lines

HTMLLoaderConfig

Used by HTMLLoader, HTMLStringLoader, and internally by WebLoader (which constructs its own instance with a fixed tag list).

FieldTypeDefaultPurpose
parserstr"html.parser"BeautifulSoup parser backend — "html.parser", "lxml", or "html5lib"
tags_to_extractOptional[List[str]]NoneIf set, only text inside these tags is extracted; None = extract all <body> text
tags_to_removeList[str]["script", "style", "nav", "footer", "header"]Tags stripped (via .decompose()) before text extraction
extract_linksboolFalseCollect {"text", "href"} for every <a href> into metadata["links"]
extract_tablesboolTrueEmit each <table> as an additional separate LoadedDocument

WebLoaderConfig

Used by WebLoader and MultiURLLoader.

FieldTypeDefaultPurpose
timeoutint10Per-request timeout in seconds
headersDict[str, str]{}Extra HTTP headers merged into the default User-Agent header
verify_sslboolTrueTLS certificate verification
max_retriesint3Total request attempts before raising ConnectionError
retry_delayfloat1.0Base delay (seconds) for exponential backoff: retry_delay * 2**attempt
encodingOptional[str]NoneForces response encoding; None = use response.apparent_encoding
extract_metadataboolTrueDeclared but not read anywhere in web_loader.py — has no effect; metadata extraction is actually controlled by the HTMLLoaderConfig used internally.

DirectoryLoaderConfig

Used by DirectoryLoader.

FieldTypeDefaultPurpose
glob_patternstr"**/*"Glob pattern for file discovery
exclude_patternsList[str]["*.pyc", "__pycache__/*", ".git/*", ".DS_Store"]fnmatch patterns checked against both the relative path and the bare filename
recursiveboolTrueUse Path.rglob() vs. Path.glob()
silent_errorsboolFalseLog and skip files that fail to load instead of raising
show_progressboolTruePrint a \r-overwriting progress line to stdout
max_concurrencyint4Thread pool size when use_multithreading=True
use_multithreadingboolTrueUse ThreadPoolExecutor instead of sequential loading

LoaderConfig

A master container holding one default instance of every sub-config above. Not currently consumed by any loader constructor — none of the loader classes accept a LoaderConfig; each loader takes its own specific *Config type directly. It exists as a convenience aggregate, e.g. for applications that want to manage one configuration object for an entire pipeline.

Python · 10 lines
@dataclass
class LoaderConfig:
    text: TextLoaderConfig
    pdf: PDFLoaderConfig
    docx: DocxLoaderConfig
    csv: CSVLoaderConfig
    json: JSONLoaderConfig
    html: HTMLLoaderConfig
    web: WebLoaderConfig
    directory: DirectoryLoaderConfig

GraphLoaderConfig and TableLoaderConfig

These live in graph_loader.py and table_loader.py respectively (not config_loader.py) and are documented alongside their loaders:

Usage pattern

Python · 9 lines
from zerda.document_loaders import CSVLoader, CSVLoaderConfig

config = CSVLoaderConfig(
    content_columns=["title", "body"],
    metadata_columns=["author", "published_date"],
    max_rows=1000,
)
loader = CSVLoader("articles.csv", config=config)
docs = loader.load()

Related pages