StructureAwareChunker
On this page
Module: chunks.structure_aware_chunker
Import: from zerda.chunks import StructureAwareChunker
Table of Contents
- Purpose
- Algorithm
- Constructor
- Public Methods
- Document Type Detection
- Section Handling Rules
- Chunk Types Produced
- Examples
- Best Practices
Purpose
StructureAwareChunker parses document structure (Markdown headings, HTML tags, paragraph breaks) and creates chunks that respect those natural boundaries. A chapter never gets split mid-paragraph if avoidable; code blocks and tables are always kept intact.
Use when:
- Input is Markdown documentation or HTML pages
- You need chunk boundaries aligned with heading/section structure
- Preserving code blocks and tables is critical
Algorithm
1. DocumentStructureParser.detect_document_type(text) → DocumentType
2. DocumentStructureParser.parse(text, doc_type) → List[StructuredSection]
3. _merge_small_sections(): merge short sections with next (unless code/table/header)
4. For each section:
- Code block? → one chunk (ChunkType.CODE_BLOCK)
- Table? → one chunk (ChunkType.TABLE)
- List? → one chunk (ChunkType.LIST_ITEM)
- Header node? → one chunk (ChunkType.HEADER)
- Normal text? → _split_large_section() → one or more SECTION chunks
5. _finalize() sets position and total_chunks
Constructor
StructureAwareChunker(
config: Optional[ChunkConfig] = None,
*,
max_chunk_size: int = 1024,
min_chunk_size: int = 50,
language: str = "auto",
split_on_headers: bool = True,
preserve_code_blocks: bool = True,
)
Parameters
Public Methods
Inherited from BaseChunker:
chunk(text, doc_id="", source="") → List[DocumentChunk]
chunk_documents(documents) → List[DocumentChunk]
Document Type Detection
DocumentStructureParser.detect_document_type() uses heuristics:
Section Handling Rules
Chunk Types Produced
Metadata Set
Each chunk gets:
metadata.section # heading text of parent section
metadata.heading_level # 0-6 (0 = body, 1 = H1, ...)
metadata.is_header # True for HEADER chunks
metadata.chunk_type # ChunkType enum value
metadata.document_type # detected DocumentType
metadata.extra # {"tag": "p"} for HTML, {"is_heading_node": True} for headers
Examples
Markdown Document
from zerda.chunks import StructureAwareChunker
md = """
# Introduction
This is the intro section.
## Installation
```bash
pip install chunks
Usage
Use the library like this.
"""
chunker = StructureAwareChunker()
chunks = chunker.chunk(md, source="docs.md")
for c in chunks:
print(c.metadata.chunk_type.value, "|", c.metadata.section, "|", c.text[:40])
HTML Document
html = """
<html><body>
<h1>Getting Started</h1>
<p>Welcome to the tutorial.</p>
<ul><li>Step one</li><li>Step two</li></ul>
</body></html>
"""
chunker = StructureAwareChunker()
chunks = chunker.chunk(html, source="page.html")
With Configuration
from zerda.chunks import StructureAwareChunker, ChunkConfig
config = ChunkConfig(
max_chunk_size=800,
min_chunk_size=100,
preserve_code_blocks=True,
split_on_headers=True,
)
chunker = StructureAwareChunker(config=config)
Disabling Code Block Preservation
# Split large code blocks like any other text
chunker = StructureAwareChunker(preserve_code_blocks=False)
Best Practices
- Always use for Markdown docs, API references, and HTML pages.
- Set
split_on_headers=True(default) to ensure each section is independently retrievable. - Inspect
metadata.sectionto filter chunks from a specific heading during retrieval. - Use
ChunkMode.HYBRIDwhen sections may be long prose that also benefits from semantic splitting.