ZerdaZerda

Tutorial 01 — Hello Chunks

On this page

Level: Beginner
Time: ~10 minutes

What You'll Learn

  • Create a ChunkManager
  • Process your first document
  • Read chunk metadata

Step 1 — Install

Bash · 3 lines
pip install numpy
# Optional for semantic features:
# pip install sentence-transformers torch

Step 2 — Your First Chunk

Python · 18 lines
from zerda.chunks import MultilanguageTextChunker

chunker = MultilanguageTextChunker()

text = """
The Python programming language was created by Guido van Rossum and 
released in 1991. Python emphasizes code readability and simplicity.
It supports multiple programming paradigms including procedural, 
object-oriented, and functional programming.

Python has a large standard library and an active community. It is 
widely used in web development, data science, artificial intelligence, 
and automation. The language uses indentation to define code blocks, 
which makes it distinctive among programming languages.
"""

chunks = chunker.chunk(text, source="python_intro.txt")
print(f"Produced {len(chunks)} chunk(s)")

Step 3 — Inspect a Chunk

Python · 14 lines
chunk = chunks[0]

print("=== Chunk ===")
print(f"ID:          {chunk.chunk_id}")
print(f"Doc ID:      {chunk.doc_id}")
print(f"Chars:       {chunk.char_count}")
print(f"Words:       {chunk.word_count}")
print(f"Type:        {chunk.metadata.chunk_type.value}")
print(f"Position:    {chunk.metadata.position} of {chunk.metadata.total_chunks}")
print(f"Score:       {chunk.metadata.score:.3f}")
print(f"Keywords:    {chunk.metadata.keywords}")
print()
print("Text preview:")
print(chunk.text[:200])

Step 4 — Serialize to JSON

Python · 4 lines
import json

data = [c.to_dict() for c in chunks]
print(json.dumps(data[0], ensure_ascii=False, indent=2))

Step 5 — What Happened?

ChunkManager in AUTO mode used StructureAwareChunker (the default). It:

  1. Detected PLAIN_TEXT document type (no Markdown/HTML markers).
  2. Split on double newlines (paragraph boundaries).
  3. Ran MetadataEnricher — added keywords and importance score.
  4. Ran Deduplicator — no duplicates found.
  5. Finalized position and total_chunks.

Related pages