ArabicEmbedder

embeddings.arabic_embedder.ArabicEmbedder(BaseEmbedder) — local sentence-transformers embedder with an Arabic-specific text-normalization pipeline.

Purpose

Runs a local multilingual (or Arabic-specific) sentence-transformers model and, before encoding, optionally applies Arabic normalization: diacritic removal, hamza unification, alef maksura normalization, tatweel removal, and (at the aggressive level) taa marbuta normalization and repeated-character collapsing.

Constructor

ArabicEmbedder(
    model_name: Optional[str] = None,
    device: Optional[str] = None,
    cache_dir: Optional[str] = None,
    normalize_embeddings: bool = True,
    batch_size: int = 32,
    enable_preprocessing: bool = True,
    max_seq_length: int = 512,
    normalization_level: str = "standard",
    track_processing_stats: bool = True,
    auto_download: bool = True,
    trust_remote_code: bool = False,
    **kwargs,
)

Parameters

RECOMMENDED_MODELS (class attribute)

Raises

• ImportError if sentence-transformers isn't installed.
• Propagates any other exception raised while loading the model.
• See Known Issues item 2 for a broken raise statement on the auto_download=False failure path.

encode()

encode(
    texts: Union[str, List[str]],
    show_progress_bar: bool = False,
    convert_to_numpy: bool = True,
    batch_size: Optional[int] = 32,
    return_valid_indices: bool = False,
    skip_preprocessing: bool = False,
    **kwargs,
) -> Union[np.ndarray, Tuple[np.ndarray, List[int]]]

Behavior

1. Empty input (not texts) returns an empty array (and [] for indices if requested) without raising.
2. str input is wrapped in a list; the function tracks single_text = True to unwrap the result at the end.
3. Non-str/non-list/tuple input raises TypeError. List/tuple elements must all be str or None, or TypeError is raised.
4. Empty/whitespace-only/None entries are filtered out and their original indices tracked in valid_indices; if none remain, an empty result is returned.
5. If enable_preprocessing and not skip_preprocessing, texts are normalized via _preprocess_texts().
6. Calls self.model.encode(...). If a RuntimeError mentions "out of memory", retries once with half the batch size.
7. For list input where some original entries were invalid, the result is expanded back to the original length with zero vectors at invalid positions.

Returns

• Single string in → 1D np.ndarray (or (np.ndarray, List[int]) if return_valid_indices=True).
• List in → 2D np.ndarray of shape (n, embedding_dim), zero-padded at positions that were filtered out.

Raises

• TypeError for invalid input types.
• Propagates any exception from the underlying model call that isn't an OOM RuntimeError.

Text normalization

_normalize_arabic(text: str) -> str

Applies normalization in tiers:

• minimal: removes diacritics (\u064B–\u065F) only.
• standard (default): minimal + normalizes hamza variants (أإآ→ا, ؤ→و, ئ→ي) + alef maksura (ى→ي) + removes tatweel (ـ).
• aggressive: standard + normalizes taa marbuta (ة→ه).

_preprocess_texts(texts: List[str]) -> List[str]

Calls _normalize_arabic per text, then (for standard/aggressive) collapses repeated punctuation (!?؟.), then (for aggressive only) collapses 3+ repeated characters to 2 via ARABIC_NORMALIZATION_PATTERNS['repeated_chars'], then collapses whitespace.

Persistence

save_embeddings(texts, filepath, save_texts=True, save_metadata=True) -> None

Encodes texts and saves to a .npz file (via np.savez) with keys embeddings, optionally texts and metadata (a dict of model/config info wrapped in a 0-d object array). Automatically strips a trailing .npz from filepath before calling np.savez (which re-appends it) to avoid a doubled extension.

• Raises ValueError if texts is empty.

load_embeddings(filepath, load_texts=False, load_metadata=False) -> Union[np.ndarray, Tuple]

Loads a .npz file saved by save_embeddings (appends .npz to filepath if missing). Returns just the embeddings array, or a tuple including texts/metadata depending on flags.

• Raises FileNotFoundError if the file doesn't exist, KeyError if the 'embeddings' key is missing from the file.

Similarity utilities

compute_similarity(text1, text2, metric='cosine') -> float

Unlike BaseEmbedder.similarity(), this implementation computes true cosine similarity (divides by both vector norms), plus 'euclidean' (1 / (1 + ||a - b||)) and 'dot' (raw dot product). Accepts either raw strings (auto-encoded) or precomputed embeddings for either argument.

• Raises ValueError for unknown metric values.

find_most_similar(query, candidates, top_k=5, metric='cosine') -> List[Tuple[int, str, float]]

Encodes the query and all candidates once, computes compute_similarity pairwise, and returns the top-k (index, text, score) tuples sorted by score descending.

Diagnostics

benchmark(sample_texts=None, num_iterations=10, warmup_iterations=2) -> Dict

Runs num_iterations timed encode passes (after warmup_iterations untimed warmup passes) over sample_texts (defaults to eight built-in Arabic sample sentences if not provided) and returns timing statistics (avg_time, std_time, min_time, max_time, median_time, texts_per_second) plus model metadata.

get_model_info() -> Dict, get_processing_stats() -> Dict[str, Any], reset_stats()

Return/reset model configuration and ArabicProcessingStats (normalization counts, diacritics removed, hamzas normalized, elapsed time) respectively. get_processing_stats() returns {'tracking_enabled': False} if track_processing_stats=False.

list_recommended_models() (staticmethod)

Returns ArabicEmbedder.RECOMMENDED_MODELS directly — the live class dict, not a copy.

Context manager

__enter__ returns self; __exit__ logs a session summary (from ArabicProcessingStats.get_summary() if tracking is enabled) and calls self.cleanup().

ArabicProcessingStats (dataclass)

Tracked fields: total_texts, normalized_texts, avg_normalization_time, removed_diacritics, normalized_hamzas, start_time. .get_summary() returns a dict including a derived normalization_rate and elapsed_time_seconds.

Examples

from zerda.embeddings import ArabicEmbedder

# Basic usage with default (multilingual-mini) model
embedder = ArabicEmbedder()
vec = embedder.encode("مرحباً بكم")

# Choosing a recommended model + aggressive normalization
embedder = ArabicEmbedder(model_name="arabert", normalization_level="aggressive")

# Finding similar texts
results = embedder.find_most_similar(
    query="الذكاء الاصطناعي",
    candidates=["تعلم الآلة", "الطبخ العربي", "الشبكات العصبية"],
    top_k=2,
)

# Save/reload embeddings
embedder.save_embeddings(["نص أول", "نص ثاني"], "my_vectors")
vectors, texts, meta = embedder.load_embeddings("my_vectors", load_texts=True, load_metadata=True)

Related APIs

• BaseEmbedder — parent class
• HuggingFaceEmbedder — sibling local embedder without Arabic-specific preprocessing