ZerdaZerda

Quickstart

On this page

Get chunking in under five minutes.

Table of Contents

Simplest Usage

Python · 4 lines
from zerda.chunks import SemanticChunker

chunker = SemanticChunker()
chunks = chunker.chunk("Your document text here.", source="doc.txt")

Arabic Text

Python · 12 lines
from zerda.chunks import ArabicTextChunker

chunker = ArabicTextChunker(
    chunk_size=512,
    overlap=100,
    use_semantic_chunking=False,   # rule-based, no GPU needed
    fix_spacing=True,
)
chunks = chunker.chunk("مرحباً بالعالم. هذا مثال على تقسيم النص العربي.", doc_id="ar_doc")

for chunk in chunks:
    print(chunk.text)

For Arabic with normalization preprocessing:

Python · 7 lines
from zerda.chunks import ArabicNormalizer, NormalizingChunkerWrapper, ArabicTextChunker

normalizer = ArabicNormalizer()
base_chunker = ArabicTextChunker(use_semantic_chunking=False)
chunker = NormalizingChunkerWrapper(base_chunker, normalizer=normalizer)

chunks = chunker.chunk("الأُسْتاذُ يُدرِّسُ اللُّغةَ العَربيَّةَ")

Multilingual Text

Python · 10 lines
from zerda.chunks import MultilanguageTextChunker

chunker = MultilanguageTextChunker(
    chunk_size=512,
    overlap=100,
    language="auto",               # auto-detects from script
    use_semantic_chunking=False,
)
chunks = chunker.chunk("Mixed language text here.")
print(f"Detected language: {chunks[0].metadata.language}")

Inspecting Chunk Metadata

Every DocumentChunk carries rich metadata:

Python · 15 lines
chunk = chunks[0]

print(chunk.chunk_id)              # UUID
print(chunk.doc_id)                # document identifier
print(chunk.char_count)            # character count
print(chunk.word_count)            # word count
print(chunk.content_hash)          # SHA-256 dedup hash
print(chunk.metadata.source)       # file/URL source
print(chunk.metadata.position)     # 0-based position in doc
print(chunk.metadata.total_chunks) # total chunks in doc
print(chunk.metadata.chunk_type)   # ChunkType enum value
print(chunk.metadata.keywords)     # list of top keywords
print(chunk.metadata.score)        # importance score 0-1
print(chunk.metadata.char_start)   # character offset in original
print(chunk.metadata.char_end)     # character offset end

Serialize to dict:

Python · 3 lines
data = chunk.to_dict()
import json
print(json.dumps(data, ensure_ascii=False, indent=2))

LangChain-Style Splitters

The library includes drop-in-compatible splitters:

Python · 22 lines
from zerda.chunks import (
    CharacterTextSplitter,
    RecursiveCharacterTextSplitter,
    TokenTextSplitter,
    SentenceTextSplitter,
)

# Character splitter
splitter = CharacterTextSplitter(separator="\n\n", chunk_size=500, chunk_overlap=50)
texts = splitter.split_text(raw_text)

# Recursive splitter (like LangChain's)
recursive = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
texts = recursive.split_text(raw_text)

# Arabic-aware recursive
arabic_recursive = RecursiveCharacterTextSplitter(arabic_mode=True, chunk_size=500, chunk_overlap=50)
texts = arabic_recursive.split_text(arabic_text)

# Token splitter (requires tiktoken)
token_splitter = TokenTextSplitter(chunk_size=256, chunk_overlap=32)
texts = token_splitter.split_text(raw_text)

All splitters support split_documents(docs) and create_documents(texts, metadatas).

Next Steps

Related pages