ZerdaZerda

ArabicNormalizer & NormalizingChunkerWrapper

On this page

Module: chunks.arabic_preprocessor
Import: from zerda.chunks import ArabicNormalizer, NormalizingChunkerWrapper, NormalizerConfig

Table of Contents

NormalizerConfig

Python · 13 lines
@dataclass
class NormalizerConfig:
    normalize_alef: bool = True
    normalize_teh_marbuta: bool = True
    normalize_yeh: bool = True
    remove_diacritics: bool = True
    remove_tatweel: bool = True
    normalize_hamza: bool = True
    remove_punctuation: bool = False
    normalize_numbers: bool = False
    max_repeated_chars: int = 2
    fix_spacing: bool = True
    language: str = "arabic"

All flags are True by default except remove_punctuation and normalize_numbers, preserving text readability while normalizing orthographic variants.

FieldEffect
normalize_alefإأآأ → ا
normalize_teh_marbutaة → ه
normalize_yehيى → ي
remove_diacriticsRemove U+064B–U+065F harakat
remove_tatweelRemove kashida ـ (U+0640)
normalize_hamzaؤئء → ء
remove_punctuationStrip .,!?،؛ and similar
normalize_numbersEastern Arabic numerals → Western (٠١٢... → 012...)
max_repeated_charsCollapse اااا → اا
fix_spacingNormalize whitespace around punctuation

ArabicNormalizer

Purpose

ArabicNormalizer is a preprocessing pipeline that standardizes Arabic text orthography before chunking. It eliminates spelling variations that would otherwise cause the same word to be treated as distinct tokens.

Typical use: wrap with NormalizingChunkerWrapper so normalization runs transparently before any chunker.

ArabicNormalizer Constructor

Python · 3 lines
ArabicNormalizer(
    config: Optional[NormalizerConfig] = None,
)

If config is None, NormalizerConfig() defaults are used.

ArabicNormalizer Methods

normalize(text: str) → str

Applies the full normalization pipeline.

Python · 3 lines
normalizer = ArabicNormalizer()
clean = normalizer.normalize("الأُسْتاذُ يُدرِّسُ اللُّغةَ")
print(clean)  # "الاستاذ يدرس اللغه"

normalize_batch(texts: List[str]) → List[str]

Applies normalize() to each text in the list.

Python
cleaned = normalizer.normalize_batch(["مرحباً", "السَّلامُ"])

get_normalization_stats(original: str, normalized: str) → dict

Returns statistics comparing before/after normalization:

Python · 6 lines
{
    "original_length": 42,
    "normalized_length": 38,
    "reduction_ratio": 0.095,
    "chars_removed": 4,
}

Normalization Pipeline

Steps execute in this exact order:

code · 10 lines
1. normalize_alef      → إأآأ → ا
2. normalize_teh_marbuta → ة → ه
3. normalize_yeh       → يى → ي
4. normalize_hamza     → ؤئ → ء
5. remove_tatweel      → remove ـ
6. remove_diacritics   → remove U+064B–U+065F harakat
7. normalize_numbers   → ٠-٩ → 0-9  (if enabled)
8. remove_punctuation  → strip .,!?،؛  (if enabled)
9. max_repeated_chars  → collapse runs
10. fix_spacing        → normalize whitespace

NormalizingChunkerWrapper

Purpose

NormalizingChunkerWrapper is a decorator that transparently applies ArabicNormalizer.normalize() to the input text before passing it to any BaseChunker. The resulting chunks contain normalized text.

NormalizingChunkerWrapper Constructor

Python · 5 lines
NormalizingChunkerWrapper(
    chunker: BaseChunker,
    normalizer: Optional[ArabicNormalizer] = None,
    config: Optional[NormalizerConfig] = None,
)
ParameterTypeDefaultDescription
chunkerBaseChunker—The wrapped chunker
normalizerArabicNormalizer | NoneNonePre-built normalizer. If None, creates one from config.
configNormalizerConfig | NoneNoneConfig for auto-created normalizer. Ignored if normalizer is provided.

NormalizingChunkerWrapper Methods

chunk(text, doc_id="", source="") → List[DocumentChunk]

  1. Calls self.normalizer.normalize(text).
  2. Passes normalized text to self.chunker.chunk(...).
  3. Returns results unchanged.

chunk_documents(documents) → List[DocumentChunk]

Normalizes each document's page_content then calls the wrapped chunker.

Examples

Normalizer Standalone

Python · 10 lines
from zerda.chunks import ArabicNormalizer, NormalizerConfig

config = NormalizerConfig(
    normalize_alef=True,
    remove_diacritics=True,
    normalize_numbers=True,
)
normalizer = ArabicNormalizer(config=config)
clean = normalizer.normalize("الأُسْتاذُ يُدرِّسُ ٣ طلاب")
print(clean)  # "الاستاذ يدرس 3 طلاب"

Wrapped Chunker

Python · 12 lines
from zerda.chunks import (
    ArabicNormalizer, NormalizingChunkerWrapper,
    ArabicTextChunker, NormalizerConfig,
)

normalizer = ArabicNormalizer()
chunker = ArabicTextChunker(chunk_size=400, use_semantic_chunking=False)
wrapper = NormalizingChunkerWrapper(chunker, normalizer=normalizer)

chunks = wrapper.chunk(raw_arabic_text, doc_id="doc_1", source="ocr_scan.pdf")
for c in chunks:
    print(c.text)  # normalized text

Custom Normalization

Python · 7 lines
config = NormalizerConfig(
    remove_diacritics=True,
    remove_punctuation=True,  # for keyword extraction
    normalize_numbers=True,
    max_repeated_chars=1,     # collapse all repetitions
)
normalizer = ArabicNormalizer(config=config)

Stats

Python · 6 lines
normalizer = ArabicNormalizer()
original = "الأُسْتاذُ الكَريمُ"
normalized = normalizer.normalize(original)
stats = normalizer.get_normalization_stats(original, normalized)
print(stats)
# {'original_length': 18, 'normalized_length': 13, 'reduction_ratio': 0.278, 'chars_removed': 5}

Wrapping MultilanguageTextChunker

Python · 7 lines
from zerda.chunks import MultilanguageTextChunker, NormalizingChunkerWrapper, ArabicNormalizer

wrapper = NormalizingChunkerWrapper(
    MultilanguageTextChunker(language="arabic"),
    normalizer=ArabicNormalizer(),
)
chunks = wrapper.chunk(text)

Related pages