MultilanguageTextChunker

Module: chunks.multi_chunker
Import: from zerda.chunks import MultilanguageTextChunker

Table of Contents

• Purpose
• Supported Languages
• Constructor
• Language Detection
• Chunking Pipeline
• Public Methods
• Examples
• Best Practices
• Common Mistakes

Purpose

MultilanguageTextChunker handles 17 languages by detecting the script/language automatically and applying language-specific tokenization, sentence splitting, and (optionally) BERT models. It unifies all language paths under a single API.

Use when:

• You receive documents in unknown or mixed languages
• You need a single chunker that handles Arabic, Chinese, Japanese, Korean, and European languages uniformly

Supported Languages

Constructor

MultilanguageTextChunker(
    chunk_size: int = 512,
    overlap: int = 128,
    min_chunk_size: int = 50,
    language: str = "auto",
    use_semantic_chunking: bool = False,
    device: Optional[str] = None,
    config: Optional[ChunkConfig] = None,
)

Parameters

Language Detection

When language="auto", _detect_language(text) applies script heuristics in priority order:

1. Arabic Unicode block → "arabic"
2. CJK Unified Ideographs → "chinese"
3. Hiragana/Katakana → "japanese"
4. Hangul syllables → "korean"
5. Cyrillic → "russian"
6. Devanagari → "hindi"
7. Stopword matching (French, German, Spanish, Portuguese, Italian, Dutch, Polish, Turkish, Vietnamese)
8. Default → "english"

The detected language is stored in chunk.metadata.language.

Chunking Pipeline

Rule-Based Path

1. _split_sentences(text, language) — uses language-appropriate delimiters.
   • CJK languages: 。！？\n
   • Arabic: .!?؟\n
   • Other: .!?\n
2. Accumulate sentences into chunks up to chunk_size.
3. Apply character overlap.
4. Drop chunks shorter than min_chunk_size.

Semantic Path

1. Same sentence splitting.
2. Load language-appropriate BERT tokenizer + model (cached per model name).
3. Compute mean-pooled embeddings.
4. Open new chunk when cosine similarity drops below 0.75.

Public Methods

chunk(text, doc_id="", source="") → List[DocumentChunk]

Main entry point. Detects language (unless forced), chunks, finalizes.

All resulting chunks have:

• metadata.language — detected or forced language string
• metadata.chunk_type — ChunkType.SEMANTIC (semantic mode) or ChunkType.SECTION (rule-based)

chunk_documents(documents) → List[DocumentChunk]

Processes a list of Document objects sequentially.

Examples

Auto-Detection

from zerda.chunks import MultilanguageTextChunker

chunker = MultilanguageTextChunker()

arabic_chunks = chunker.chunk("مرحباً بالعالم")
print(arabic_chunks[0].metadata.language)   # "arabic"

chinese_chunks = chunker.chunk("你好，世界！这是一个测试。")
print(chinese_chunks[0].metadata.language)  # "chinese"

english_chunks = chunker.chunk("Hello world. This is a test.")
print(english_chunks[0].metadata.language)  # "english"

Forced Language

chunker = MultilanguageTextChunker(language="french", chunk_size=300)
chunks = chunker.chunk(french_text, source="article.txt")

Semantic Mode

chunker = MultilanguageTextChunker(
    language="auto",
    use_semantic_chunking=True,
    chunk_size=512,
    overlap=100,
)
chunks = chunker.chunk(multilingual_doc)

Batch Processing

from chunks import MultilanguageTextChunker, Document

chunker = MultilanguageTextChunker()
docs = [
    Document(page_content="English text here.", metadata={"source": "en.txt"}),
    Document(page_content="نص عربي هنا.", metadata={"source": "ar.txt"}),
    Document(page_content="中文文本在这里。", metadata={"source": "zh.txt"}),
]
all_chunks = chunker.chunk_documents(docs)

for c in all_chunks:
    print(c.metadata.language, "|", c.text[:30])

Best Practices

• Use language="auto" (default) unless you know the document language exactly — the heuristic is fast and reliable for clear scripts.
• For multilingual documents (mixed Arabic/English), pass language="multilingual" to use XLM-RoBERTa.
• In semantic mode, BERT models are cached after first load — the first call per language is slow, subsequent calls are fast.

Common Mistakes

Related Pages

• ArabicTextChunker →
• SemanticChunker →
• ChunkConfig →
• EmbeddingProvider →