TextLoader & MarkdownLoader

Module: text_loader · Extends BaseFileLoader

TextLoader

Overview

Loads plain text files (.txt, .text, .log, .rst) with automatic encoding fallback. Use this for any unstructured plain-text source — logs, README files, raw transcripts.

Signature

class TextLoader(BaseFileLoader):
    SUPPORTED_EXTENSIONS = [".txt", ".text", ".log", ".rst"]

    def __init__(
        self,
        file_path: str,
        config: Optional[TextLoaderConfig] = None,
        encoding: str = "utf-8",
        autodetect_encoding: bool = True,
    ): ...

    def load(self) -> List[LoadedDocument]: ...

Parameters

Returns

load() returns a single-element list — the entire file as one LoadedDocument. There is no per-line or per-paragraph splitting in this loader (use load_and_split() for chunking).

Raises

• FileNotFoundError / ValueError — from BaseFileLoader._validate_file() at construction time (missing file, wrong extension)
• Any decode error is caught internally and handled via the fallback chain below — load() itself should not raise UnicodeDecodeError under normal conditions since latin-1 never fails to decode arbitrary bytes

Internal workflow / algorithm — _read_file()

1. Try reading with self.config.encoding and self.config.errors (default "replace").
2. On UnicodeDecodeError or LookupError:
   • If autodetect_encoding is False, re-raise.
   • Otherwise, attempt chardet.detect() on the raw bytes (only if chardet is installed — silently skipped otherwise).
   • If detection succeeds and yields a different encoding, retry the read with it.
   • As a final fallback, read with latin-1, which is guaranteed to succeed on any byte sequence (it maps every byte 0–255 to a code point).

Metadata produced

In addition to the base file metadata (source, loader_type, loaded_at, file_name, file_type, file_size_bytes):

Examples

Basic:

from zerda.document_loaders import TextLoader

docs = TextLoader("notes.txt").load()
print(docs[0].page_content[:100])

Disabling autodetection (strict mode):

loader = TextLoader("ascii_only.txt", autodetect_encoding=False, encoding="ascii")
docs = loader.load()  # raises UnicodeDecodeError if the file isn't valid ASCII

With explicit config:

from zerda.document_loaders import TextLoader, TextLoaderConfig

config = TextLoaderConfig(encoding="windows-1256", errors="ignore")
docs = TextLoader("legacy_arabic.txt", config=config).load()

Arabic/RTL text: no special handling is required — TextLoader reads bytes and decodes them; RTL rendering is a downstream display concern, not something this loader transforms.

Best practices

• Leave autodetect_encoding=True (the default) when ingesting files from unknown or mixed sources.
• Install chardet if your corpus has non-UTF-8 files; without it, the loader silently skips straight to the latin-1 fallback, which can produce mojibake for non-Latin encodings (e.g. Windows-1256 Arabic text decoded as latin-1 will be garbled, not an error).

Common mistakes

• Assuming load() splits the file into multiple documents — it does not; it's always exactly one LoadedDocument per file.
• Forgetting that latin-1 fallback never raises, so silent encoding mismatches can pass through undetected — check metadata["encoding_used"] if data quality matters.

MarkdownLoader

Overview

Loads Markdown files (.md, .markdown, .mdown, .mkd) with optional syntax stripping, code-block removal, and automatic title extraction from the first # Heading.

Signature

class MarkdownLoader(BaseFileLoader):
    SUPPORTED_EXTENSIONS = [".md", ".markdown", ".mdown", ".mkd"]

    def __init__(
        self,
        file_path: str,
        encoding: str = "utf-8",
        strip_markdown: bool = False,
        remove_code_blocks: bool = False,
        autodetect_encoding: bool = True,
    ): ...

    def load(self) -> List[LoadedDocument]: ...

Note: unlike most other loaders, MarkdownLoader does not accept a config object — all options are direct constructor kwargs.

Parameters

Returns

A single-element list, same as TextLoader.

Internal workflow

1. Read raw text (_read_file() — chardet → latin-1 fallback on UnicodeDecodeError).
2. Extract title from the first line starting with "# " (_extract_title).
3. If remove_code_blocks=True: regex-strip ```...``` and ~~~...~~~ blocks.
4. If strip_markdown=True: run _strip_markdown() (see algorithm below) on the (possibly code-block-stripped) text.
5. Build metadata and wrap in a LoadedDocument.

Algorithm — _strip_markdown()

A sequence of regex substitutions, applied in this order:

1. Remove heading markers (^#{1,6}\s+)
2. Remove bold/italic markers (*/_ wrapping)
3. Convert [text](url) links to bare text
4. Remove image syntax ![alt](url) entirely
5. Unwrap inline code `code` → code
6. Remove horizontal rules (---, ***, ___)
7. Remove blockquote markers (> )
8. Remove unordered (-/*/+) and ordered (1.) list markers
9. Collapse 3+ consecutive newlines to 2

This is a lossy, regex-based transformation — it is not a full Markdown AST parser, so deeply nested or unusual Markdown constructs may not be fully stripped.

Metadata produced

[!WARNING] heading_count caveat heading_count is computed as raw_text.count('\n#'), which counts every newline immediately followed by # — this includes ##, ### etc. (each counted once per line, correctly), but will under-count a heading on the very first line of the file (no preceding \n) and can be thrown off by # characters in code blocks or inline text that happen to start a line.

Examples

Basic:

from zerda.document_loaders import MarkdownLoader

docs = MarkdownLoader("README.md").load()
print(docs[0].metadata["title"])

Clean prose for embedding (strip syntax, drop code):

loader = MarkdownLoader("blog_post.md", strip_markdown=True, remove_code_blocks=True)
docs = loader.load()
# docs[0].page_content now reads as plain prose, no headings/links/code

Keep raw Markdown (default) for a Markdown-aware downstream chunker:

docs = MarkdownLoader("docs/guide.md").load()

Best practices

• Use strip_markdown=True when feeding content into embedding models that don't benefit from Markdown syntax noise.
• Use remove_code_blocks=True for natural-language search/RAG use cases where code snippets would pollute semantic search; keep code blocks (False) for technical documentation Q&A where users might ask about code.

Common mistakes

• Expecting MarkdownLoader to return separate documents per heading/section — it returns one document for the whole file; split sections yourself or use load_and_split() with a Markdown-aware splitter.
• Relying on heading_count for anything precise — see the caveat above.

Related APIs

• LoadedDocument
• TextLoaderConfig
• DirectoryLoader — dispatches to TextLoader/MarkdownLoader automatically by extension
• AutoLoader