ZerdaZerda

TableLoader, TableLoaderConfig, InMemoryTableLoader

On this page

Module: table_loader — the Table Reader

[!NOTE] Bilingual source Like graph_loader, this module's comments and docstrings are written primarily in Arabic. This page documents the described behavior in English.

Overview

A format-agnostic table extractor: pulls tabular data out of Excel, CSV/TSV, PDF, HTML, and Markdown files, and renders each table as a LoadedDocument in your choice of output format (Markdown, CSV, JSON, or plain text) — either one document per whole table, or one document per row.

TableLoader is not wired into EXTENSION_MAP/AutoLoader/DirectoryLoader and must be used explicitly. It overlaps in scope with CSVLoader/ExcelLoader (for CSV/Excel) and HTMLLoader's table extraction (for HTML), but is distinguished by its uniform multi-format output formatting and explicit per-table vs. per-row document granularity control.

TableLoaderConfig

Python · 12 lines
@dataclass
class TableLoaderConfig:
    output_format: str = "markdown"          # markdown | csv | json | text
    per_row: bool = False
    sheet_names: Optional[List[str]] = None
    skip_empty_tables: bool = True
    max_rows_per_doc: Optional[int] = None
    include_header_in_rows: bool = True
    column_separator: str = " | "
    pdf_extraction_method: str = "pdfplumber"  # pdfplumber | camelot
    html_parser: str = "html.parser"
    encoding: str = "utf-8"
FieldTypeDefaultPurpose
output_formatstr"markdown"One of "markdown", "csv", "json", "text" — controls how a whole table is rendered into page_content
per_rowboolFalseTrue = one LoadedDocument per row; False = one per whole table
sheet_namesOptional[List[str]]NoneRestrict Excel loading to named sheets; None = all sheets
skip_empty_tablesboolTrueSkip tables with zero data rows
max_rows_per_docOptional[int]NoneTruncate rows included in a whole-table document (does not apply to per_row mode, where every row already gets its own document)
include_header_in_rowsboolTrueIn per_row mode, prefix each cell with its column header ("col: value") rather than just joining raw values
column_separatorstr" | "Used for output_format="text" and for per_row mode when include_header_in_rows=False
pdf_extraction_methodstr"pdfplumber""pdfplumber" or "camelot"
html_parserstr"html.parser"BeautifulSoup backend
encodingstr"utf-8"Text encoding

TableLoader

Extends BaseFileLoader

Signature

Python · 17 lines
class TableLoader(BaseFileLoader):
    SUPPORTED_EXTENSIONS = [
        ".xlsx", ".xls", ".xlsm", ".csv", ".tsv", ".pdf", ".html", ".htm", ".md", ".markdown",
    ]

    def __init__(
        self,
        file_path: str,
        config: Optional[TableLoaderConfig] = None,
        *,
        output_format: str = "markdown",
        per_row: bool = False,
        encoding: str = "utf-8",
    ): ...

    def load(self) -> List[LoadedDocument]: ...
    def lazy_load(self) -> Iterator[LoadedDocument]: ...

[!WARNING] output_format/per_row always override config Same pattern as GraphLoader: __init__ unconditionally does self.config.output_format = output_format and self.config.per_row = per_row after setting self.config, regardless of whether a fully-customized config= was passed. To get a non-default output_format/per_row while also passing a custom config, you must set them as direct keyword arguments too — values inside a passed-in config object for these two specific fields are always overwritten.

Parameters

NameTypeDefaultDescription
file_pathstrrequiredPath to the table-bearing file
configOptional[TableLoaderConfig]NoneSee above
output_formatstr (keyword-only)"markdown"Always applied — see warning
per_rowbool (keyword-only)FalseAlways applied — see warning
encodingstr"utf-8"Text encoding

Returns

List[LoadedDocument] — count and shape depend on per_row and the number of tables/sheets/pages found.

Raises

  • FileNotFoundError / ValueError — at construction
  • ImportError — openpyxl (Excel), pdfplumber or camelot-py (PDF, depending on pdf_extraction_method), beautifulsoup4 (HTML) — each with a clear install message
  • ValueError — unrecognized extension reaching lazy_load() (defensive)

Internal workflow — format dispatch

Python · 6 lines
ext = self.file_path.suffix.lower()
if ext in (".xlsx", ".xls", ".xlsm"):   yield from self._load_excel()
elif ext in (".csv", ".tsv"):            yield from self._load_csv(delimiter=...)
elif ext == ".pdf":                       yield from self._load_pdf()
elif ext in (".html", ".htm"):             yield from self._load_html()
elif ext in (".md", ".markdown"):           yield from self._load_markdown()

Format: Excel — _load_excel()

Opens with openpyxl.load_workbook(read_only=True, data_only=True). Iterates config.sheet_names or all sheets; for each, reads all rows via iter_rows(values_only=True), treats the first row as headers, the rest as data, and builds a _RawTable per sheet → _table_to_docs().

Format: CSV/TSV — _load_csv(delimiter)

Reads the whole file as a single table (first row = headers, rest = data rows) via csv.reader. Unlike CSVLoader, this treats the entire CSV as one table object rather than one document per row by default (though per_row=True produces the same per-row granularity).

Format: PDF — _load_pdf()

Dispatches to _pdf_pdfplumber() or _pdf_camelot() based on config.pdf_extraction_method:

  • pdfplumber: iterates pages, calls page.extract_tables(), skips tables with fewer than 2 rows (tbl[0] = header, rest = data); each table gets page_number and an incrementing table_idx in metadata.
  • camelot: camelot.read_pdf(path, pages="all"); each detected table's pandas DataFrame has its first row treated as headers; metadata includes camelot's own accuracy score (a model-reported extraction confidence metric) and page (camelot reports this on the table object itself, e.g. as a string like "3").

Format: HTML — _load_html()

For each <table>: headers from <thead><tr> if present, else falls back to treating the first <tbody> row as headers (headers = cells when row_idx == 0 and no <thead> headers were found). Skips empty tables if skip_empty_tables.

Format: Markdown — _load_markdown()

Parses GitHub-Flavored-Markdown tables via the static method _parse_markdown_tables(text), which uses a regex (\|.+\|\n\|[-:| ]+\|\n(?:\|.+\|\n)*) to find header-row + separator-row + data-rows blocks, splitting each |-delimited line into cells.

[!NOTE] Markdown table regex limitations The regex requires lines to end with \n (so a table at the very end of a file without a trailing newline may not match) and expects a strict | cell | cell | structure per line; tables using alternate Markdown table syntaxes (e.g. without leading/trailing pipes) will not be detected.

_RawTable (internal dataclass)

Python · 5 lines
@dataclass
class _RawTable:
    headers: List[str]
    rows: List[List[str]]
    metadata: Dict[str, Any] = field(default_factory=dict)

An internal intermediate representation, not part of the public API, used to decouple "got this table from somewhere" from "render this table into a LoadedDocument".

_table_to_docs(table) → one or more LoadedDocument

  • If config.skip_empty_tables and the table has no rows: yields nothing.
  • If config.per_row: delegates to _per_row_docs() (one document per row).
  • Else: delegates to _whole_table_doc() (one document for the entire table).

_whole_table_doc(table) — output formatting

Renders the entire table (respecting max_rows_per_doc as a truncation limit) using one of four formatters based on config.output_format:

output_formatMethodOutput shape
"markdown"_to_markdownGFM-style | col | col | table with a --- separator row
"csv"_to_csv_textStandard CSV text via csv.writer into a StringIO buffer
"json"_to_json_textPretty-printed JSON array of row objects (dict(zip(headers, row)))
"text" (default for any unrecognized value)_to_plain_textLines joined by config.column_separator

_per_row_docs(table) — per-row formatting

For each row: if include_header_in_rows, renders as "header1: val1 | header2: val2 | ..."; otherwise just joins raw values with column_separator. Each row gets its own LoadedDocument with row_index in metadata.

Metadata produced

Varies by format and granularity, but generally includes: source, file_format ("excel", "csv", "pdf", "html", "markdown"), sheet_name/page_number/table_index (as applicable), total_rows, total_cols, loader_type: "TableLoader", output_format (or "row" in per-row mode), and row_index (per-row mode only).

Complexity

  • Time: O(rows × columns) per table, summed across all tables/sheets/pages.
  • Space: Each format builds the full table in memory (_RawTable) before rendering — there's no row-level streaming within a single table's extraction, though lazy_load() does yield each table's document(s) as soon as that table is processed, before moving to the next sheet/page.

Thread safety

No shared mutable state; safe across concurrent instances on different files.

Examples

Basic — all Excel tables as Markdown:

Python · 4 lines
from zerda.document_loaders import TableLoader

docs = TableLoader("data.xlsx").load()
print(docs[0].page_content)  # GFM markdown table

Per-row extraction, plain text format:

Python · 6 lines
from zerda.document_loaders import TableLoader, TableLoaderConfig

config = TableLoaderConfig(per_row=True, output_format="text")
loader = TableLoader("report.pdf", config=config)
for doc in loader.lazy_load():
    print(doc.page_content)

JSON output for downstream structured processing:

Python · 3 lines
docs = TableLoader("inventory.csv", output_format="json").load()
import json
records = json.loads(docs[0].page_content)

Specific Excel sheets only:

Python · 2 lines
config = TableLoaderConfig(sheet_names=["Q1", "Q2"])
docs = TableLoader("annual.xlsx", config=config).load()

PDF tables via camelot instead of pdfplumber:

Python · 2 lines
config = TableLoaderConfig(pdf_extraction_method="camelot")
docs = TableLoader("financial_statement.pdf", config=config).load()

Truncate huge tables to a manageable document size:

Python · 2 lines
config = TableLoaderConfig(max_rows_per_doc=200)
docs = TableLoader("big_export.xlsx", config=config).load()

Error handling — missing PDF table extraction dependency:

Python · 4 lines
try:
    docs = TableLoader("report.pdf").load()
except ImportError as e:
    print(f"Install pdfplumber: {e}")

Production usage — Arabic table data, per-row for RAG with row-level citations:

Python · 5 lines
config = TableLoaderConfig(per_row=True, include_header_in_rows=True)
docs = TableLoader("بيانات.xlsx", config=config).load()
for doc in docs:
    # doc.page_content e.g. "الاسم: أحمد | العمر: 30 | المدينة: القاهرة"
    index_for_retrieval(doc)

Best practices

  • Use per_row=True when you need row-level retrieval granularity (e.g. "find the row about product X"); use the default whole-table mode when the table's meaning depends on seeing all rows together (e.g. a small comparison table).
  • For PDF tables, try pdfplumber first (the default, lighter dependency); fall back to camelot only if extraction quality is insufficient, since camelot has heavier dependencies (OpenCV via [cv] extras).
  • Always pass output_format/per_row as direct keyword arguments to TableLoader(...), not only inside a config= object, per the override warning above.

Common mistakes

  • Passing TableLoaderConfig(output_format="json") via config= without also passing output_format="json" directly — it gets silently reset to "markdown".
  • Expecting max_rows_per_doc to limit per_row mode — it only affects whole-table document rendering; per-row mode inherently produces one document per row with no built-in cap (use skip_rows/row-count logic upstream if you need to limit total rows processed).
  • Assuming the Markdown table parser handles every valid Markdown table syntax variant — see the regex limitation note above.

InMemoryTableLoader

Extends BaseDocumentLoader directly

Overview

Loads a table passed directly as Python lists — no file required.

Signature

Python · 11 lines
class InMemoryTableLoader(BaseDocumentLoader):
    def __init__(
        self,
        headers: List[str],
        rows: List[List[str]],
        config: Optional[TableLoaderConfig] = None,
        source_name: str = "in_memory_table",
    ): ...

    def load(self) -> List[LoadedDocument]: ...
    def lazy_load(self) -> Iterator[LoadedDocument]: ...

Parameters

NameTypeDefaultDescription
headersList[str]requiredColumn names
rowsList[List[str]]requiredData rows
configOptional[TableLoaderConfig]None → TableLoaderConfig()Honored as given, no constructor-kwarg override (unlike TableLoader)
source_namestr"in_memory_table"Logical source identifier

Internal workflow

lazy_load() constructs an uninitialized TableLoader instance via TableLoader.__new__(TableLoader) (bypassing __init__ entirely, since there's no real file to validate) and manually sets just the two attributes (config, file_path) that _table_to_docs() needs, then reuses that method directly. This is a deliberate code-reuse technique to avoid duplicating the formatting logic (_to_markdown, _to_csv_text, etc.) between TableLoader and InMemoryTableLoader.

Examples

Basic:

Python · 7 lines
from zerda.document_loaders import InMemoryTableLoader

loader = InMemoryTableLoader(
    headers=["الاسم", "العمر", "المدينة"],
    rows=[["أحمد", "30", "القاهرة"], ["سارة", "25", "الإسكندرية"]],
)
docs = loader.load()

Programmatically-generated table (e.g. from a database query):

Python · 5 lines
from zerda.document_loaders import InMemoryTableLoader, TableLoaderConfig

rows = [[str(r["id"]), r["name"], str(r["revenue"])] for r in query_results]
config = TableLoaderConfig(output_format="markdown")
docs = InMemoryTableLoader(headers=["id", "name", "revenue"], rows=rows, config=config).load()

Per-row, for individual record retrieval:

Python · 2 lines
config = TableLoaderConfig(per_row=True)
docs = InMemoryTableLoader(headers=["sku", "qty"], rows=inventory_rows, config=config).load()

Best practices

  • Use this when table data already exists as in-memory Python structures (query results, API responses) rather than round-tripping through a temporary CSV file and TableLoader.

Common mistakes

  • Mismatched row lengths vs. headers — the formatting methods (_to_markdown, etc.) generally pad short rows but this is on a best-effort basis per formatter; verify your row lengths match your header count for predictable output.

Related pages