ZerdaZerda

Hybrid Search

On this page

Overview

HybridSearchRAG fuses semantic search (delegated to the wrapped rag_system) with keyword search (BM25 or TF-IDF, computed locally) using one of several fusion strategies, and can compare all methods side-by-side.

SearchConfig

Python · 8 lines
@dataclass
class SearchConfig:
    semantic_weight: float = 0.7
    keyword_weight: float = 0.3
    min_score: float = 0.3
    top_k: int = 10
    enable_reranking: bool = True
    fusion_method: str = "weighted_sum"

StanzaTokenizer

Python · 4 lines
class StanzaTokenizer:
    def __init__(self, language: str = "ar", use_lemmatization: bool = True): ...
    def tokenize(self, text: str) -> List[str]: ...
    def tokenize_batch(self, texts: List[str]) -> List[List[str]]: ...

Wraps the Stanza NLP library for tokenization/lemmatization, with a _simple_tokenize fallback (used, per the code structure, when Stanza isn't available or fails) — this makes both BM25Scorer and TFIDFScorer functional even without the Stanza dependency installed, at reduced linguistic accuracy.

BM25Scorer

Python · 5 lines
class BM25Scorer:
    def __init__(self, k1: float = 1.5, b: float = 0.75, language: str = "ar", use_lemmatization: bool = True): ...
    def fit(self, documents: List[str]): ...
    def score(self, query: str, document: str, doc_idx: int) -> float: ...
    def batch_score(self, query: str, documents: List[str]) -> List[float]: ...

Standard Okapi BM25: k1 controls term-frequency saturation (typical 1.2–2.0), b controls length normalization (typical ~0.75). Per its docstring, BM25 is preferred here over TF-IDF for short/medium texts. fit() must be called (indexing document frequencies/lengths) before score/batch_score.

TFIDFScorer

Python · 5 lines
class TFIDFScorer:
    def __init__(self, language: str = "ar", use_lemmatization: bool = True): ...
    def fit(self, documents: List[str]): ...
    def score(self, query: str, document: str) -> float: ...
    def batch_score(self, query: str, documents: List[str]) -> List[float]: ...

Simpler classic TF-IDF cosine-style scorer; use when BM25's extra tuning parameters aren't needed.

HybridSearchRAG

Constructor

Python · 8 lines
def __init__(
    self,
    rag_system,
    config: Optional[SearchConfig] = None,
    keyword_method: str = "bm25",
    language: str = "ar",
    use_lemmatization: bool = True,
)

keyword_method is "bm25" or "tfidf".

train_keyword_index()

Must be called before keyword_search/hybrid_search — fits the chosen scorer (BM25Scorer.fit or TFIDFScorer.fit) over the backend's document set.

Search methods

MethodReturnsDescription
semantic_search(query, top_k=None) -> List[Tuple]Vector-only results delegated to rag_system.
keyword_search(query, top_k=None) -> List[Tuple]BM25/TF-IDF-only results.
hybrid_search(query, top_k=None, semantic_weight=None, keyword_weight=None) -> List[Tuple[DocumentChunk, float]]Combines both via the configured fusion method.
compare_methods(query, top_k=5) -> DictRuns all three and returns their results side-by-side for evaluation.

Fusion strategies (_fuse_results dispatches to)

MethodAlgorithm
_weighted_sum_fusionsem_weight * semantic_score + key_weight * keyword_score, using _normalize_scores to map each score list to [0, 1] first.
_reciprocal_rank_fusionRRF: sum(1 / (k + rank)) across result lists, k defaulting to 60 — chosen, per the docstring, because it needs no score normalization.
_max_fusionSelects the higher of the two scores per document.

fusion_method in SearchConfig selects which of these hybrid_search uses internally.

generate(query, search_method="hybrid", include_sources=False) -> str

Runs the selected search method, formats sources via _format_sources if requested, and generates an answer.

get_stats() -> dict

System statistics (e.g. whether the keyword index has been trained, corpus size).

Async API

Python · 5 lines
async def asemantic_search(self, query, top_k=None)
async def akeyword_search(self, query, top_k=None)
async def ahybrid_search(self, query, top_k=None, semantic_weight=None, keyword_weight=None)
async def agenerate(self, query, search_method="hybrid", include_sources=False) -> str
async def astream(self, query, search_method="hybrid")

ahybrid_search explicitly runs semantic and keyword searches in parallel (per its docstring) rather than sequentially, unlike the sync hybrid_search. Plus __aenter__/__aexit__.

Complete example

Python · 12 lines
from zerda.rag.types.hybrid_search import HybridSearchRAG, SearchConfig

config = SearchConfig(semantic_weight=0.6, keyword_weight=0.4, fusion_method="rrf", top_k=8)
search = HybridSearchRAG(backend, config=config, keyword_method="bm25", language="en")
search.train_keyword_index()

results = search.hybrid_search("machine learning applications in healthcare")
for chunk, score in results:
    print(score, chunk)

comparison = search.compare_methods("machine learning applications in healthcare")
print(comparison.keys())  # e.g. semantic / keyword / hybrid buckets

Best practices

  • Re-run train_keyword_index() whenever documents are added/removed from the backend — the keyword index is a separate, locally-fit structure that does not auto-sync with the vector DB.
  • Prefer Reciprocal Rank Fusion (_reciprocal_rank_fusion) when semantic and keyword score distributions aren't directly comparable, since it sidesteps normalization entirely.

Common mistakes

  • Calling keyword_search/hybrid_search before train_keyword_index() — the scorer has no fitted vocabulary yet.
  • Assuming hybrid_search parallelizes semantic and keyword search like its async counterpart does — the synchronous version does not (see ahybrid_search's docstring for the explicit contrast).
  • Graph RAG — has its own enable_hybrid_search config flag for a related but separately-implemented mode.

Related pages