CSVLoader & ExcelLoader

Module: csv_loader · Extends BaseFileLoader

CSVLoader

Overview

Loads CSV/TSV files, producing one LoadedDocument per row by default, with flexible control over which columns become content vs. metadata.

Signature

class CSVLoader(BaseFileLoader):
    SUPPORTED_EXTENSIONS = [".csv", ".tsv"]

    def __init__(
        self,
        file_path: str,
        config: Optional[CSVLoaderConfig] = None,
        content_columns: Optional[List[str]] = None,
        metadata_columns: Optional[List[str]] = None,
        source_column: Optional[str] = None,
        encoding: str = "utf-8",
        delimiter: str = ",",
    ): ...

    def load(self) -> List[LoadedDocument]: ...
    def lazy_load(self) -> Iterator[LoadedDocument]: ...

Parameters

Returns

load() returns list(self.lazy_load()). Rows that produce empty content (after filtering to content_columns and stripping) are silently omitted — not returned as empty documents.

Raises

• FileNotFoundError / ValueError — at construction
• UnicodeDecodeError — only if both the configured encoding and the latin-1 fallback fail (practically near-impossible, since latin-1 decodes any byte sequence) — included for completeness and to surface a clear final error message rather than an unbounded retry loop

Internal workflow / algorithm — lazy_load()

[!TIP] Implementation note The source documents a fix here: the original implementation caught UnicodeDecodeError and called lazy_load() recursively after mutating self.config.encoding — which could silently loop or yield duplicate documents if the retry also failed. The current implementation iterates a fixed two-item list [configured_encoding, "latin-1"] with no recursion, return-ing immediately on first success.

encodings_to_try = [self.config.encoding, "latin-1"]
for enc in encodings_to_try:
    try:
        yield from self._iter_rows(enc)
        return
    except UnicodeDecodeError:
        continue
raise UnicodeDecodeError(...)  # only if both attempts failed

_iter_rows(encoding):

1. Open the file with newline="" (required by the csv module) and the given encoding, errors="strict".
2. Skip config.skip_rows raw lines via next(f, None).
3. Parse with csv.DictReader(f, delimiter=..., quotechar=...).
4. For each row (up to config.max_rows if set), build a LoadedDocument via _row_to_document(); yield it if non-empty.

_row_to_document(row, row_number):

• If content_columns is set: content = row_joiner.join(f"{col}: {value}") for each listed column with a non-empty value.
• Otherwise: content = row_joiner.join(f"{k}: {v}") for every column with a non-empty value.
• Returns None (row skipped) if resulting content is empty after stripping.
• Metadata always includes row_number (1-indexed, relative to the parsed data rows). If metadata_columns is set, those raw column values are added. Else if content_columns is set, every column not in content_columns is added to metadata automatically. If source_column is set and present in the row, it overrides metadata["source"].

Metadata produced

Base file metadata + row_number, plus whatever metadata_columns/the content-columns-complement rule above adds.

Complexity

• Time: O(n) in number of rows, single pass.
• Space: O(1) per yielded document in lazy_load() — true streaming; O(n) if you call load(), which materializes the full list.

Thread safety

A single CSVLoader instance is not designed for concurrent lazy_load() iteration from multiple threads (the underlying file handle is not shared-safe), but separate instances/files are independent.

Examples

Basic — one document per row, all columns:

from zerda.document_loaders import CSVLoader

docs = CSVLoader("contacts.csv").load()
print(docs[0].page_content)  # "name: Alice\nemail: alice@example.com\n..."

Selecting content vs. metadata columns:

from zerda.document_loaders import CSVLoader, CSVLoaderConfig

config = CSVLoaderConfig(
    content_columns=["title", "body"],
    metadata_columns=["author", "published_date"],
)
docs = CSVLoader("articles.csv", config=config).load()

TSV with header skip and row limit:

config = CSVLoaderConfig(skip_rows=1, max_rows=500)
docs = CSVLoader("export.tsv", config=config).load()

Streaming large files without loading everything into memory:

loader = CSVLoader("huge_dataset.csv")
for doc in loader.lazy_load():
    process(doc)  # one row in memory at a time

Error handling:

try:
    docs = CSVLoader("data.csv", encoding="ascii").load()
except UnicodeDecodeError as e:
    print(f"Both ascii and latin-1 failed: {e}")

Best practices

• Use lazy_load() for very large CSVs instead of load() to avoid materializing the full document list in memory.
• Set content_columns explicitly when your CSV has many columns you don't want polluting embeddings (e.g. internal IDs).

Common mistakes

• Setting delimiter="," explicitly on a .tsv file — this disables the automatic tab-delimiter override, since the override only triggers when delimiter is left at its literal default value ",".
• Forgetting that empty rows (after content-column filtering) are silently dropped — row counts in your output may not match the source file's row count.

ExcelLoader

Overview

Loads Excel workbooks (.xlsx, .xls, .xlsm), producing one LoadedDocument per data row, across one or all sheets.

Signature

class ExcelLoader(BaseFileLoader):
    SUPPORTED_EXTENSIONS = [".xlsx", ".xls", ".xlsm"]

    def __init__(
        self,
        file_path: str,
        sheet_name: Optional[str] = None,
        load_all_sheets: bool = False,
        content_columns: Optional[List[str]] = None,
        metadata_columns: Optional[List[str]] = None,
        skip_rows: int = 0,
        max_rows: Optional[int] = None,
    ): ...

    def load(self) -> List[LoadedDocument]: ...

Note: ExcelLoader does not use CSVLoaderConfig or any dataclass config — all options are plain constructor kwargs.

Parameters

Returns

List[LoadedDocument], one per non-empty data row across the selected sheet(s).

Raises

• ImportError — "openpyxl is required..." for .xlsx/.xlsm if openpyxl isn't installed
• ImportError — "pandas is required for .xls files..." for legacy .xls if pandas/xlrd aren't installed
• ValueError — if a requested sheet_name doesn't exist (lists available sheet names in the message)

Internal workflow

def load(self):
    ext = self.file_path.suffix.lower()
    if ext == ".xls":
        return self._load_with_pandas()
    # .xlsx / .xlsm path:
    wb = openpyxl.load_workbook(..., read_only=True, data_only=True)
    sheet_names = (all sheets, or [sheet_name], or [first sheet])
    for sname in sheet_names:
        documents.extend(self._load_sheet(wb[sname], sname))
    wb.close()

[!TIP] Implementation note The source docstring notes that .xls support via the xlrd engine (through pandas) and .xlsm support via openpyxl were added on top of the original .xlsx-only implementation.

.xlsx/.xlsm — _load_sheet(ws, sheet_name): reads all rows via ws.iter_rows(values_only=True); the first row becomes headers (missing header cells become col_{i}); applies skip_rows and max_rows to the remaining data rows; builds a "key: value" text per row (filtered to content_columns if set), skipping rows whose resulting content is empty.

.xls — _load_with_pandas(): uses pd.ExcelFile(path, engine="xlrd"), iterates with df.iterrows(); filters out NaN-valued cells (compared as the literal string "nan"); same content/metadata column logic; row_number metadata is i + 2 (accounting for the header row and 1-indexing, matching spreadsheet row numbers as a human would see them).

Metadata produced

sheet_name, row_number (spreadsheet-style: data row 1 → metadata row_number=2, accounting for the header), plus any metadata_columns values and base file metadata.

Complexity

• Time: O(rows × columns) per sheet.
• Space: read_only=True mode in openpyxl streams rows from the file rather than loading the whole workbook into memory, but _load_sheet still calls list(ws.iter_rows(...))-equivalent iteration and accumulates all resulting LoadedDocuments in a list before returning — so overall memory is O(total non-empty rows across loaded sheets).

Examples

Basic — first sheet only:

from zerda.document_loaders import ExcelLoader

docs = ExcelLoader("sales_data.xlsx").load()

All sheets:

docs = ExcelLoader("workbook.xlsx", load_all_sheets=True).load()
print({d.metadata["sheet_name"] for d in docs})

Specific sheet with column selection:

docs = ExcelLoader(
    "report.xlsx",
    sheet_name="Q4 Summary",
    content_columns=["Product", "Revenue"],
    metadata_columns=["Region"],
).load()

Legacy .xls:

docs = ExcelLoader("legacy_export.xls").load()  # requires pandas + xlrd

Error handling — bad sheet name:

try:
    docs = ExcelLoader("data.xlsx", sheet_name="DoesNotExist").load()
except ValueError as e:
    print(e)  # "Sheet 'DoesNotExist' not found. Available: ['Sheet1', 'Sheet2']"

Best practices

• Prefer .xlsx over .xls where possible — the .xls path requires the extra pandas/xlrd dependency chain and is generally slower.
• Use load_all_sheets=True deliberately, not by default, on workbooks with many sheets — it multiplies the document count and load time.

Common mistakes

• Assuming row numbers in metadata are zero-indexed — they follow spreadsheet conventions (row_number=2 is the first data row, since row 1 is the header).
• Not installing xlrd for .xls files and assuming openpyxl alone covers all Excel formats — it doesn't; openpyxl cannot read the legacy binary .xls format.

Related APIs

• CSVLoaderConfig
• TableLoader — alternative table-oriented extraction for Excel with Markdown/CSV/JSON output formats and whole-table (not just per-row) documents
• LoadedDocument

Version History

Cannot be determined from the repository, beyond the documented additions of .xls (via pandas/xlrd) and .xlsm (via openpyxl) support noted in ExcelLoader.load()'s docstring.