ZerdaZerda

DocumentStructureParser

On this page

Module: chunks.document_structure_parser
Import: from zerda.chunks import DocumentStructureParser, StructuredSection

Table of Contents

Purpose

DocumentStructureParser converts raw text (Markdown, HTML, or plain text) into a list of StructuredSection objects, each representing a logical document unit: heading, paragraph, code block, table, list, etc.

It is used internally by StructureAwareChunker but can be used standalone.

StructuredSection

Python · 10 lines
@dataclass
class StructuredSection:
    content: str
    level: int = 0              # Heading level (0=body, 1=H1, 2=H2, ...)
    section_type: str = "text"  # "heading", "code", "table", "list", "text"
    is_code_block: bool = False
    is_table: bool = False
    is_list: bool = False
    parent_heading: str = ""    # Text of parent heading section
    extra: Dict[str, Any] = field(default_factory=dict)

Fields

FieldTypeDescription
contentstrRaw text content of this section
levelintHeading depth (0 = body text, 1 = H1, etc.)
section_typestrOne of: "heading", "code", "table", "list", "text"
is_code_blockboolTrue if content is a fenced code block
is_tableboolTrue if content is an HTML table
is_listboolTrue if content is an HTML list
parent_headingstrText of the nearest ancestor heading
extradictParser-specific extras: {"tag": "p"} for HTML, {"is_heading_node": True} for headings

Constructor

Python
DocumentStructureParser()

No parameters. Instantiate once and reuse across multiple parse() calls.

Public Methods

detect_document_type

Python
def detect_document_type(self, text: str) -> DocumentType

Detects the document type from text heuristics.

Returns: DocumentType enum value — HTML, MARKDOWN, ARABIC, or PLAIN_TEXT.

SignalReturns
<html>, <div>, <p>, <h1> in textDocumentType.HTML
# at line start, ```, **bold**, [link]DocumentType.MARKDOWN
Arabic character ratio > 20%DocumentType.ARABIC
DefaultDocumentType.PLAIN_TEXT

parse

Python · 5 lines
def parse(
    self,
    text: str,
    doc_type: Optional[DocumentType] = None,
) -> List[StructuredSection]

Parses text into a list of StructuredSection objects.

If doc_type is None, detect_document_type() is called automatically.

Returns: List[StructuredSection] — may be empty for empty input.

Markdown Parsing

  1. Scans lines sequentially.
  2. Detects fenced code blocks (```) — wraps entire block as is_code_block=True.
  3. Detects headings (# , ## , etc.) — wraps as section_type="heading" with level=count("#").
  4. Accumulates remaining lines into paragraph sections.
  5. Splits large paragraph blocks on blank lines.

HTML Parsing

If beautifulsoup4 is installed:

  • Uses BeautifulSoup(text, "html.parser").
  • Walks the DOM: extracts <h1>–<h6> as headings, <p> as text, <pre>/<code> as code blocks, <table> as tables, <ul>/<ol> as lists.

If beautifulsoup4 is not installed:

  • Falls back to regex tag stripper: removes all HTML tags, then treats result as plain text.

Plain Text / Arabic Parsing

Splits on double newlines into paragraphs, wraps each as a StructuredSection with section_type="text".

Examples

Basic Markdown Parsing

Python · 11 lines
from zerda.chunks import DocumentStructureParser

parser = DocumentStructureParser()

md = """
# Introduction
Welcome to the guide.

## Installation
```bash
pip install chunks"""
code · 3 lines
sections = parser.parse(md)
for s in sections:
    print(s.section_type, s.level, repr(s.content[:40]))

Expected output:

code · 4 lines
heading 1 '# Introduction'
text 0 'Welcome to the guide.'
heading 2 '## Installation'
code 0 '```bash\npip install chunks\n```'

Type Detection

Python · 7 lines
from zerda.chunks import DocumentStructureParser

parser = DocumentStructureParser()
print(parser.detect_document_type("<html><body>...</body></html>"))   # DocumentType.HTML
print(parser.detect_document_type("# Title\n## Section"))            # DocumentType.MARKDOWN
print(parser.detect_document_type("مرحباً بالعالم"))                  # DocumentType.ARABIC
print(parser.detect_document_type("Plain text."))                     # DocumentType.PLAIN_TEXT

HTML Parsing

Python · 11 lines
html = """
<html><body>
<h1>Getting Started</h1>
<p>Welcome to the tutorial.</p>
<pre><code>print("hello")</code></pre>
</body></html>
"""

sections = parser.parse(html)
for s in sections:
    print(s.section_type, s.is_code_block, s.content[:30])

Forced Type

Python · 3 lines
# Force plain-text mode even for HTML-looking content
from zerda.chunks import DocumentType
sections = parser.parse(text, doc_type=DocumentType.PLAIN_TEXT)

Related pages