ZerdaZerda

HTMLLoader & HTMLStringLoader

On this page

Module: html_loader

HTMLLoader

Extends BaseFileLoader

Overview

Loads .html/.htm/.xhtml files, extracting clean text via BeautifulSoup along with page metadata (title, meta description, Open Graph tags, language) and optionally links and tables as separate documents.

Signature

Python · 11 lines
class HTMLLoader(BaseFileLoader):
    SUPPORTED_EXTENSIONS = [".html", ".htm", ".xhtml"]

    def __init__(
        self,
        file_path: str,
        config: Optional[HTMLLoaderConfig] = None,
        encoding: str = "utf-8",
    ): ...

    def load(self) -> List[LoadedDocument]: ...

Parameters

NameTypeDefaultDescription
file_pathstrrequiredPath to the HTML file
configOptional[HTMLLoaderConfig]NoneSee HTMLLoaderConfig
encodingstr"utf-8"File encoding

Returns

List[LoadedDocument] — at least one document (the main page text) plus one additional document per <table> found, if config.extract_tables is True (the default).

Raises

  • FileNotFoundError / ValueError — at construction
  • ImportError — "beautifulsoup4 is required..." if bs4 isn't installed
  • IOError — "Cannot read HTML file: ..." wrapping any exception from reading the file

Internal workflow — _parse_html(html, source)

  1. Parse with BeautifulSoup(html, config.parser).
  2. Extract page metadata from <head> before removing any tags (_extract_page_metadata): <title>, <meta name="description"|"author"|"keywords">, Open Graph tags (og:title, og:description, og:url → stored as og_title etc.), and <html lang>/xml:lang.
  3. Remove unwanted tags (default: script, style, nav, footer, header) via .decompose().
  4. Extract text:
    • If config.tags_to_extract is set: concatenate get_text() from each matching tag, joined by "\n\n".
    • Else: extract all text from <body> (or the whole soup if no <body>), joined by "\n".
  5. Collapse 3+ consecutive newlines to 2.
  6. Build the main document's metadata, merging in the page metadata from step 2 plus char_count.
  7. If config.extract_links: collect {"text", "href"} for every <a href> into metadata["links"] of the main document.
  8. If config.extract_tables: call _extract_tables() and append each table's LoadedDocument to the result list.

_extract_tables(soup, source)

For each <table> (in document order), builds rows by joining cell text (<td>/<th>) with " | ", skipping rows where every cell is empty. Skips the whole table if it produces no non-empty rows. Each table document's metadata includes element: "table" and table_index (0-indexed).

Metadata produced (main document)

title, description, author, keywords (if present in <meta>), og_title/og_description/og_url (if present), language (if lang attribute found), char_count, links (if extract_links=True), plus base file metadata.

Examples

Basic:

Python · 4 lines
from zerda.document_loaders import HTMLLoader

docs = HTMLLoader("index.html").load()
print(docs[0].metadata["title"])

Extract specific content regions only:

Python · 4 lines
from zerda.document_loaders import HTMLLoader, HTMLLoaderConfig

config = HTMLLoaderConfig(tags_to_extract=["article", "main"])
docs = HTMLLoader("blog_post.html", config=config).load()

Disable table extraction (treat as plain text only):

Python · 3 lines
config = HTMLLoaderConfig(extract_tables=False)
docs = HTMLLoader("page.html", config=config).load()
assert len(docs) == 1

Extract links for a crawler-style pipeline:

Python · 4 lines
config = HTMLLoaderConfig(extract_links=True)
docs = HTMLLoader("sitemap_page.html", config=config).load()
for link in docs[0].metadata["links"]:
    print(link["href"])

Production usage — keep only article body, drop nav/footer/aside noise:

Python · 4 lines
config = HTMLLoaderConfig(
    tags_to_remove=["script", "style", "nav", "footer", "header", "aside", "form"],
)
docs = HTMLLoader("article.html", config=config).load()

Best practices

  • Use tags_to_extract to scope extraction to known content containers (<article>, <main>, a specific id/class wrapper if you pre-filter the HTML) when source pages have heavy chrome/boilerplate.
  • Leave extract_tables=True (default) for documents where tabular data carries meaningful information — each table becomes its own retrievable unit, which is usually better for RAG than embedding it inline with surrounding prose.

Common mistakes

  • Expecting tags_to_remove to also apply when tags_to_extract is set — removal happens before extraction regardless, so it's always in effect, but if your tags_to_extract list includes a tag that's also slated for removal (unlikely in practice, but possible with custom configs), the removal happens first and nothing will be found for that tag.
  • Forgetting that table documents are separate LoadedDocuments, not embedded in the main document's text — if you only consume docs[0], you'll miss table content.

HTMLStringLoader

Extends BaseDocumentLoader directly (not BaseFileLoader — there is no file to validate).

Overview

Loads HTML from an in-memory string rather than a file. Useful for content fetched from APIs, scraped via another tool, or generated dynamically — anywhere you have HTML text but not (and don't want) a file on disk.

Signature

Python · 9 lines
class HTMLStringLoader(BaseDocumentLoader):
    def __init__(
        self,
        html_content: str,
        source: str = "html_string",
        config: Optional[HTMLLoaderConfig] = None,
    ): ...

    def load(self) -> List[LoadedDocument]: ...

Parameters

NameTypeDefaultDescription
html_contentstrrequiredRaw HTML markup
sourcestr"html_string"Identifier stored in metadata["source"] — typically a URL or logical name
configOptional[HTMLLoaderConfig]NoneSame config type as HTMLLoader

Internal workflow

Delegates entirely to a private internal helper class _HTMLParser (a subclass of HTMLLoader that bypasses file validation by calling BaseDocumentLoader.__init__ directly and setting self.file_path = None), reusing HTMLLoader._parse_html() for all actual parsing logic. This avoids duplicating the extraction algorithm between file-based and string-based HTML loading.

Returns

Same shape as HTMLLoader.load() — main text document plus table documents if configured.

Examples

Basic:

Python · 4 lines
from zerda.document_loaders import HTMLStringLoader

html = "<html><body><h1>Hello</h1><p>World</p></body></html>"
docs = HTMLStringLoader(html, source="https://example.com").load()

Processing scraped content (this is exactly how WebLoader uses it internally):

Python · 9 lines
import requests
from zerda.document_loaders import HTMLStringLoader, HTMLLoaderConfig

response = requests.get("https://example.com/article")
docs = HTMLStringLoader(
    response.text,
    source=response.url,
    config=HTMLLoaderConfig(extract_tables=True),
).load()

Thread safety

Both classes hold no shared mutable state across instances; parsing a string or file is self-contained per call. Safe to use concurrently across different instances (e.g. via DirectoryLoader or a custom thread pool).

.

Related pages