HuggingFaceEmbedder

embeddings.hugginface_embedder.HuggingFaceEmbedder(BaseEmbedder) — local sentence-transformers embedder, no Arabic-specific text preprocessing.

[!NOTE] Module filename This class lives in hugginface_embedder.py (missing an h), but is re-exported correctly as HuggingFaceEmbedder from embeddings/__init__.py, so from embeddings import HuggingFaceEmbedder is unaffected.

Purpose

Runs any sentence-transformers-compatible model locally, with a built-in table of recommended multilingual/Arabic models, TTL-aware caching, and retry-on-failure model loading.

Constructor

HuggingFaceEmbedder(
    model_name: str = "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
    device: Optional[str] = None,
    normalize_embeddings: bool = True,
    batch_size: int = 32,
    max_length: Optional[int] = None,
    cache_embeddings: bool = True,
    cache_ttl: Optional[int] = None,
    show_progress: bool = False,
    max_retries: int = 3,
    trust_remote_code: bool = False,
    **kwargs,
)

Parameters

ARABIC_MODELS (class attribute)

A Dict[str, ModelInfo] describing five recommended models (paraphrase-multilingual-mpnet-base-v2, paraphrase-multilingual-MiniLM-L12-v2, multilingual-e5-large, bge-m3, gte-multilingual-base) with dimensions, max tokens, Arabic quality rating, download size, and description. Used only by get_model_info() to look up metadata for the currently configured model — not used to validate or resolve model_name at construction time (unlike ArabicEmbedder.RECOMMENDED_MODELS, short keys here are not accepted as model_name shortcuts).

Raises

• ImportError if sentence-transformers isn't installed.
• RuntimeError (chained from the original exception) if model loading fails after max_retries retry attempts (via the retry_on_failure decorator).

retry_on_failure(max_retries=3, delay=1.0, backoff=2.0) (module-level decorator factory)

Wraps a function to retry it on any exception, sleeping delay * backoff**attempt seconds between attempts, re-raising the last exception if all attempts fail. Used to decorate _load_model.

encode()

encode(
    texts: Union[str, List[str]],
    show_progress_bar: Optional[bool] = None,
    convert_to_numpy: bool = True,
    normalize: Optional[bool] = None,
    **kwargs,
) -> np.ndarray

Behavior

1. Raises ValueError if texts is empty/falsy.
2. Single string is wrapped in a list (is_single tracked for unwrapping at the end).
3. Raises ValueError if any element isn't a str, or if all elements are empty after .strip().
4. Iterates in batch_size-sized chunks, checking the cache (_get_from_cache) per text and only calling self.model.encode() for cache misses (with normalize_embeddings=False passed to the underlying model — normalization is applied once, afterward, on the whole array).
5. New embeddings are added to the cache (_add_to_cache), which also timestamps them for TTL expiry checks.
6. If should_normalize (from the normalize argument, defaulting to self.normalize_embeddings), L2-normalizes the stacked array (emb / (norm + 1e-10)).
7. Updates self._stats (an EmbeddingStats instance, not a dict — see BaseEmbedder → get_stats).

Returns

np.ndarray of shape (n, dim) — note that, unlike ArabicEmbedder/OllamaEmbedder, this method does not unwrap single-string input back to 1D; is_single is computed but never used to reshape the return value.

Raises

ValueError for empty/invalid input as above; propagates whatever SentenceTransformer.encode() raises for encoding failures.

Cache helpers

• _generate_cache_key(text) — returns the raw text as the key if ≤100 chars, else an MD5 hex digest.
• _is_cache_valid(cache_key) — always True if cache_ttl is falsy; otherwise checks elapsed time against _cache_timestamps.
• _get_from_cache(text) / _add_to_cache(text, embedding) — read/write helpers that also update self._stats.cache_hits / cache_misses.

Diagnostics

get_model_info() -> Dict[str, Any]

Looks up self.model_name (by substring match or exact full_name match) against ARABIC_MODELS; if found, returns that entry's fields (full_name, dimensions, max_tokens, arabic_quality, size, description). If not found, returns a fallback dict with arabic_quality: "unknown" and description: "Custom model".

get_stats() -> Dict[str, Any]

Returns self._stats.to_dict() (an EmbeddingStats.to_dict() call), including a derived cache_hit_rate percentage.

clear_cache() -> int

Clears both self._cache and self._cache_timestamps; returns the number of items removed.

ModelInfo (dataclass)

Fields: full_name, dimensions, max_tokens, arabic_quality (an ArabicQuality enum value), size, description.

EmbeddingStats (dataclass)

Fields: total_encodings, total_texts, total_time, cache_hits, cache_misses. Computed properties avg_time_per_text and cache_hit_rate (percentage). .to_dict() rounds total_time to 2 and avg_time_per_text to 4 decimal places.

ArabicQuality (enum)

EXCELLENT, GOOD, FAIR, UNKNOWN — string-valued ("excellent", "good", "fair", "unknown").

Examples

from zerda.embeddings import HuggingFaceEmbedder

embedder = HuggingFaceEmbedder()  # MiniLM, 384-dim
vec = embedder.encode("Hello world")
print(vec.shape)  # (1, 384) -- 2D even for single string, see note in encode() above

# A stronger multilingual model with a 1-hour cache TTL
embedder = HuggingFaceEmbedder(
    model_name="BAAI/bge-m3",
    cache_ttl=3600,
)

Related APIs

• BaseEmbedder — parent class
• ArabicEmbedder — sibling local embedder with Arabic-specific preprocessing