ZerdaZerda

EmbeddingProvider & cosine_similarity_matrix

On this page

Module: chunks.embeddings
Import: from zerda.chunks import EmbeddingProvider, cosine_similarity_matrix

Table of Contents

EmbeddingProvider

Purpose

EmbeddingProvider is a thin, caching wrapper around sentence-transformers (or a TF-IDF hash fallback) that provides sentence embeddings to SemanticChunker, ArabicTextChunker, MultilanguageTextChunker, and Deduplicator.

It maintains a global per-model cache (plain dict) to avoid re-computing embeddings for repeated sentences.

Constructor

Python · 6 lines
EmbeddingProvider(
    model_name: str = "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2",
    device: Optional[str] = None,
    batch_size: int = 64,
    cache_size: int = 10_000,
)
ParameterTypeDefaultDescription
model_namestrmultilingual MiniLMsentence-transformers model identifier
devicestr | NoneNoneTorch device. Auto-detected: CUDA → MPS → CPU
batch_sizeint64Sentences per encode call
cache_sizeint10_000Maximum cached entries before oldest entries are evicted

Lazy loading: The model is loaded on first call to embed() or embed_batch(), not at construction time.

Methods

embed(text: str) → np.ndarray

Returns a 1-D numpy array (embedding vector) for a single text.

Python · 3 lines
provider = EmbeddingProvider()
vec = provider.embed("Hello world")
print(vec.shape)   # (384,) for MiniLM-L12

Cache hit: returns cached vector immediately.
Cache miss: encodes via model, stores in cache, returns vector.

embed_batch(texts: List[str]) → np.ndarray

Returns a 2-D array (N, D) of embeddings for a list of texts.

  • Separates cache hits from misses.
  • Encodes only the missed texts, in batch_size sub-batches.
  • Merges cached and new embeddings in original order.
Python · 2 lines
vecs = provider.embed_batch(["Sentence one.", "Sentence two."])
print(vecs.shape)   # (2, 384)

clear_cache() → None

Empties the embedding cache.

Python
provider.clear_cache()

Fallback Behaviour

If sentence-transformers is not installed, EmbeddingProvider automatically falls back to TF-IDF hash embeddings:

  • Tokenizes text into words.
  • Hashes each word and maps to a 384-dimensional vector.
  • Normalizes the vector to unit length.

The fallback is transparent: embed() and embed_batch() return the same shape arrays. Cosine similarity computations work identically. Semantic quality is lower but the library remains functional.

Caching

The cache is a module-level dictionary keyed by (model_name, text). It is shared across all EmbeddingProvider instances that use the same model name. Cache eviction uses a simple FIFO approach when len(cache) >= cache_size.

cosine_similarity_matrix

Python · 3 lines
def cosine_similarity_matrix(
    embeddings: np.ndarray,
) -> np.ndarray

Computes pairwise cosine similarities between all row vectors in a 2-D array.

Parameters

NameTypeDescription
embeddingsnp.ndarray shape (N, D)Matrix of N embedding vectors of dimension D

Returns

np.ndarray shape (N, N) — symmetric matrix where result[i][j] is the cosine similarity between embeddings[i] and embeddings[j].

Algorithm

Python · 4 lines
# Internally:
norms = np.linalg.norm(embeddings, axis=1, keepdims=True)
normalized = embeddings / np.maximum(norms, 1e-10)
return normalized @ normalized.T

Zero-norm vectors are handled safely via np.maximum(..., 1e-10).

Complexity

DimensionValue
TimeO(N² × D)
SpaceO(N²)

Example

Python · 7 lines
import numpy as np
from zerda.chunks import cosine_similarity_matrix

vecs = np.random.rand(5, 384)
sim = cosine_similarity_matrix(vecs)
print(sim.shape)     # (5, 5)
print(sim[0, 0])     # ~1.0 (self-similarity)

Full Examples

Basic Embedding

Python · 5 lines
from zerda.chunks import EmbeddingProvider

provider = EmbeddingProvider()
v = provider.embed("The cat sat on the mat.")
print(type(v), v.shape)

Batch + Similarity

Python · 13 lines
from zerda.chunks import EmbeddingProvider, cosine_similarity_matrix

sentences = [
    "Machine learning is a subset of AI.",
    "Deep learning uses neural networks.",
    "The weather is nice today.",
]
provider = EmbeddingProvider()
vecs = provider.embed_batch(sentences)
sim = cosine_similarity_matrix(vecs)

print(f"ML↔DL similarity: {sim[0,1]:.3f}")   # high
print(f"ML↔Weather sim:   {sim[0,2]:.3f}")   # low

GPU Acceleration

Python
provider = EmbeddingProvider(device="cuda", batch_size=128)

Cache Management

Python · 3 lines
provider = EmbeddingProvider(cache_size=50_000)
# ... process documents ...
provider.clear_cache()  # free memory between batches

Related pages