ArabicTextChunker

Module: chunks.arabic_chunker
Import: from zerda.chunks import ArabicTextChunker

Table of Contents

• Purpose
• Features
• Constructor
• Public Methods
• Text Preprocessing
• Chunking Modes
• Class Methods
• Utility Methods
• Examples
• Best Practices

Purpose

ArabicTextChunker is a specialized chunker for Arabic and mixed Arabic/Latin text. It handles Arabic-specific orthographic conventions — Hamza variants, Teh Marbuta, diacritics — and supports both fast rule-based and BERT-based semantic chunking using a CAMeL-Lab Arabic model.

Use when:

• Input is Arabic or mixed Arabic/English text
• You need Arabic-aware sentence splitting (., !, ?, ؟, \n)
• Semantic chunking for Arabic with a language-specific BERT model is desired

Features

• Automatic spacing normalization (Arabic punctuation, Latin/Arabic boundaries)
• Arabic text normalization (Alef variants, Teh Marbuta, diacritics removal)
• Sentence-based rule-based chunking with hard-split fallback
• Optional BERT-based semantic chunking with cosine similarity
• Smart overlap: selects overlap sentences by semantic similarity
• Strict size enforcement with tolerance
• Parameters validated at construction

Constructor

ArabicTextChunker(
    chunk_size: int = 512,
    overlap: int = 128,
    min_chunk_size: int = 50,
    model_name: str = "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
    use_semantic_chunking: bool = False,
    device: Optional[str] = None,
    preserve_formatting: bool = True,
    fix_spacing: bool = True,
    strict_size_limit: bool = True,
    size_tolerance: float = 0.05,
    use_smart_overlap: bool = True,
    smart_overlap_threshold: float = 0.7,
)

Parameters

Validation

Raises ValueError for:

• chunk_size <= 0
• overlap < 0
• overlap >= chunk_size
• min_chunk_size <= 0 or min_chunk_size > chunk_size

Public Methods

chunk(text, doc_id="", source="") → List[DocumentChunk]

Unified entry point. Satisfies both BaseChunker and the legacy ChunkingStrategy interface.

Preprocessing applied before chunking:

1. _fix_text_spacing() if fix_spacing=True
2. _normalize_text() — Alef normalization, Teh Marbuta, diacritics removal

Text Preprocessing

Spacing Normalization (fix_spacing=True)

Text Normalization (_normalize_text())

Chunking Modes

Rule-Based (default)

1. Split on sentence delimiters: ., !, ?, ؟, \n
2. Sub-split long parts on ,, ؛, ;, ،
3. Accumulate sentences up to effective_max_size
4. Apply standard or smart overlap
5. Hard-split chunks still exceeding limit

Semantic (BERT)

1. Same sentence splitting
2. Compute BERT embeddings per sentence
3. Open new chunk when cosine_similarity < smart_overlap_threshold or size exceeded
4. Smart overlap selects top-5 most similar sentences from previous chunk

Class Methods

get_available_parameters() → List[str]

Returns the list of constructor parameter names. Useful for introspection.

params = ArabicTextChunker.get_available_parameters()

create_safely(**kwargs) → ArabicTextChunker

Filters unknown kwargs and logs a warning instead of raising TypeError.

chunker = ArabicTextChunker.create_safely(
    chunk_size=512,
    unknown_param="ignored",   # warning logged, not raised
)

Utility Methods

get_arabic_ratio(text: str) → float

Returns the ratio of Arabic characters to total characters.

ratio = chunker.get_arabic_ratio("Hello مرحبا")  # ~0.45

is_arabic_dominant(text: str, threshold: float = 0.3) → bool

Returns True when Arabic character ratio ≥ threshold.

chunker.is_arabic_dominant("مرحبا بالعالم")  # True

Examples

Rule-Based (No GPU)

from zerda.chunks import ArabicTextChunker

chunker = ArabicTextChunker(
    chunk_size=400,
    overlap=80,
    use_semantic_chunking=False,
    fix_spacing=True,
)
chunks = chunker.chunk(
    "مرحباً بالعالم. هذا مثال على تقسيم النص العربي الطويل.",
    doc_id="ar_01",
    source="doc.txt",
)

Semantic with Arabic BERT

from zerda.chunks import ArabicTextChunker, ChunkConfig

config = ChunkConfig(
    arabic="CAMeL-Lab/bert-base-arabic-camelbert-mix",
    use_semantic_chunking=True,
    chunk_size=512,
    overlap=100,
)
chunker = ArabicTextChunker(
    chunk_size=config.chunk_size,
    overlap=config.overlap,
    model_name=config.arabic,
    use_semantic_chunking=True,
)
chunks = chunker.chunk(arabic_document, doc_id="ar_doc")

Safe Construction

settings = {"chunk_size": 512, "overlap": 100, "gpu_id": 0}  # gpu_id unsupported
chunker = ArabicTextChunker.create_safely(**settings)

Normalization + Chunking

from zerda.chunks import ArabicNormalizer, NormalizingChunkerWrapper, ArabicTextChunker

normalizer = ArabicNormalizer()
base = ArabicTextChunker(use_semantic_chunking=False)
wrapped = NormalizingChunkerWrapper(base, normalizer)

chunks = wrapped.chunk("الأُسْتاذُ يُدرِّسُ اللُّغةَ العَربيَّةَ", "doc_1")

Language Detection

ratio = chunker.get_arabic_ratio("This has بعض Arabic in it")
print(f"Arabic ratio: {ratio:.0%}")
print(f"Dominant: {chunker.is_arabic_dominant('مرحبا بالعالم')}")

Best Practices

• Use use_semantic_chunking=False for CPU-only environments; rule-based is fast and accurate for Arabic.
• Always enable fix_spacing=True for OCR-extracted or web-scraped Arabic text.
• Use create_safely() when building chunkers from user-provided config dictionaries.
• For pure Arabic corpora, set model_name="CAMeL-Lab/bert-base-arabic-camelbert-mix" for best semantic quality.

Common Mistakes

Related Pages

• ArabicNormalizer →
• NormalizingChunkerWrapper →
• MultilanguageTextChunker →
• ChunkConfig →