JSONLoader & JSONLinesLoader

Module: json_loader · Extends BaseFileLoader

JSONLoader

Overview

Loads .json files containing either a single object, an array of objects, or arbitrarily nested structures navigable via a jq-style path expression. Each top-level record (after optional jq_schema extraction) becomes one LoadedDocument.

Signature

class JSONLoader(BaseFileLoader):
    SUPPORTED_EXTENSIONS = [".json"]

    def __init__(
        self,
        file_path: str,
        config: Optional[JSONLoaderConfig] = None,
        content_key: Optional[str] = None,
        metadata_func: Optional[Callable[[Dict], Dict]] = None,
        jq_schema: Optional[str] = None,
        encoding: str = "utf-8",
    ): ...

    def load(self) -> List[LoadedDocument]: ...

Parameters

Returns

List[LoadedDocument] — one per record, skipping records that produce empty content.

Raises

• FileNotFoundError / ValueError — at construction
• json.JSONDecodeError — if the file is not valid JSON (not caught internally)

Internal workflow / algorithm

1. Parse the whole file with json.load().
2. If config.jq_schema is set, apply it via _apply_jq().
3. Normalize to a list of records:
   • dict → [the_dict] (single record)
   • list → used as-is (each element is a record)
   • any scalar (e.g. a bare string/number after jq_schema extraction) → [{"value": scalar}]
4. For each record, call _record_to_document(record, index); collect non-None results.

_apply_jq(data, schema): tries import jq; jq.first(schema, data) first. If the jq package isn't installed, falls back to _simple_path() — a minimal dot-notation traversal (e.g. ".items.text") that handles dict key access and, for list-valued intermediate steps, maps the next key across all list elements (collecting non-None results). This fallback is not a full jq implementation — it doesn't support filters, array indexing syntax ([0]), pipes, or jq functions; only plain dotted key paths.

_record_to_document(record, index):

• If record is a dict:
  • If content_key is set and present → content = str(record[content_key]).
  • Elif config.text_content is True (default) → content = _dict_to_text(record) (each key rendered as "key: value", nested dict/list values JSON-serialized inline, joined by newlines).
  • Else → content = json.dumps(record, ensure_ascii=False, indent=2) (pretty JSON).
• If record is not a dict → content = str(record).
• Returns None if content.strip() is empty.
• Metadata: base file metadata + record_index; if metadata_func is callable and record is a dict, its return value (if a dict) is merged in.

Metadata produced

record_index (0-indexed position in the normalized records list) + base file metadata + anything metadata_func returns.

Side effects

metadata_func is an arbitrary user-supplied callable invoked once per dict record — any side effects it has (logging, external calls) are the caller's responsibility; the loader itself performs no I/O beyond reading the JSON file.

Examples

Basic — array of objects:

from zerda.document_loaders import JSONLoader

# data.json: [{"text": "..."}, {"text": "..."}]
docs = JSONLoader("data.json", content_key="text").load()

With a metadata extraction function:

docs = JSONLoader(
    "news.json",
    content_key="article_body",
    metadata_func=lambda r: {"title": r.get("title"), "date": r.get("date")},
).load()

Nested data via jq_schema (with the jq package installed):

from zerda.document_loaders import JSONLoader

# {"response": {"items": [{"text": "a"}, {"text": "b"}]}}
docs = JSONLoader("nested.json", jq_schema=".response.items[]", content_key="text").load()

Fallback dot-path (no jq installed):

docs = JSONLoader("nested.json", jq_schema=".response.items.text").load()
# Falls back to _simple_path — does not support the [] syntax above

Error handling — malformed JSON:

import json
try:
    docs = JSONLoader("broken.json").load()
except json.JSONDecodeError as e:
    print(f"Invalid JSON: {e}")

Best practices

• Install the jq package if you need real jq syntax (filters, array slicing, select(), etc.) — the built-in fallback is intentionally minimal.
• Use content_key whenever your records have a clear single text field; reserve text_content=True (whole-record stringification) for exploratory/ad-hoc ingestion where you don't yet know the schema.

Common mistakes

• Writing jq-array-index syntax (.items[0]) and expecting the fallback dot-path parser to support it — it doesn't; install the jq package for that.
• Forgetting errors="replace" means malformed byte sequences in the source file are silently replaced with � rather than raising — validate encoding upstream if data fidelity matters.

JSONLinesLoader

Overview

Loads JSONL/NDJSON files — one JSON object per line, the common format for ML datasets (Hugging Face, OpenAI fine-tuning exports, etc.). Streams line-by-line rather than parsing the whole file as one JSON document.

Signature

class JSONLinesLoader(BaseFileLoader):
    SUPPORTED_EXTENSIONS = [".jsonl", ".ndjson", ".jl"]

    def __init__(
        self,
        file_path: str,
        content_key: Optional[str] = None,
        metadata_func: Optional[Callable[[Dict], Dict]] = None,
        encoding: str = "utf-8",
        skip_invalid: bool = True,
        max_records: Optional[int] = None,
    ): ...

    def load(self) -> List[LoadedDocument]: ...
    def lazy_load(self) -> Iterator[LoadedDocument]: ...

Note: no config dataclass — JSONLinesLoader does not use JSONLoaderConfig; all options are direct kwargs.

Parameters

Returns

load() returns list(self.lazy_load()).

Raises

• FileNotFoundError / ValueError — at construction
• json.JSONDecodeError — only if skip_invalid=False and a line fails to parse

Internal workflow — lazy_load()

1. Open the file and iterate lines with enumerate(f, start=1) for 1-indexed line numbers.
2. Skip blank lines (after .strip()).
3. Stop if max_records reached.
4. Parse each line with json.loads(). On failure: skip-and-warn if skip_invalid, else re-raise.
5. Extract content: content_key lookup if set and the record is a dict; else join all key-value pairs as text; else str(record) for non-dict records.
6. Skip if content is empty after stripping.
7. Build metadata (line_number + base file metadata + metadata_func output) and yield the LoadedDocument.

Metadata produced

line_number (1-indexed, counting blank lines and invalid-but-skipped lines in the running count since line_num comes from enumerate over all lines, not just successfully parsed ones), plus base file metadata and metadata_func output.

Complexity

• Time: O(n) lines, single pass.
• Space: O(1) per yielded document via lazy_load() — true streaming, well-suited to multi-gigabyte JSONL datasets.

Examples

Basic:

from zerda.document_loaders import JSONLinesLoader

docs = JSONLinesLoader("dataset.jsonl", content_key="text").load()

Streaming a large dataset:

loader = JSONLinesLoader("huge_corpus.jsonl", content_key="text", max_records=10_000)
for doc in loader.lazy_load():
    embed_and_store(doc)

Strict mode — fail on any malformed line:

loader = JSONLinesLoader("dataset.jsonl", skip_invalid=False)
try:
    docs = loader.load()
except Exception as e:
    print(f"Malformed line: {e}")

With metadata extraction (e.g. Hugging Face-style fine-tuning data):

docs = JSONLinesLoader(
    "training_data.jsonl",
    content_key="completion",
    metadata_func=lambda r: {"prompt": r.get("prompt")},
).load()

Production usage — capped ingestion with error tracking:

loader = JSONLinesLoader("raw_export.jsonl", skip_invalid=True, max_records=50_000)
docs = loader.load()
# Check application logs for "Skipping invalid JSON at line N" warnings

Best practices

• Always use lazy_load() for datasets too large to comfortably fit in memory — this is the loader's primary advantage over JSONLoader.
• Leave skip_invalid=True for noisy real-world datasets; switch to False only when validating data quality, since silent skipping can mask systematic corruption.

Common mistakes

• Using JSONLoader on a .jsonl file by mistake — json.load() will fail since the file isn't a single valid JSON document (it's many, newline-separated). Use JSONLinesLoader for that format.
• Assuming line_number in metadata corresponds 1:1 with successfully-loaded document index — blank and invalid lines are counted in the line numbering but don't produce documents, so there can be gaps.

Related APIs

• JSONLoaderConfig
• LoadedDocument
• DirectoryLoader — dispatches .json → JSONLoader and .jsonl/.ndjson → JSONLinesLoader automatically