ZerdaZerda

MetadataEnricher

On this page

Module: chunks.metadata_enricher
Import: from zerda.chunks import MetadataEnricher

Table of Contents

Purpose

MetadataEnricher is a post-processing step that runs after any chunker produces raw DocumentChunk objects. It adds:

  • Keywords — top-N terms per chunk via TF-IDF
  • Importance score — 0–1 float indicating chunk relevance/richness
  • Keyword density — keyword count / total word count ratio

These fields are used by ChunkManager in its pipeline before deduplication.

Constructor

Python · 8 lines
MetadataEnricher(
    config: Optional[ChunkConfig] = None,
    *,
    extract_keywords: bool = True,
    max_keywords: int = 10,
    compute_scores: bool = True,
    header_score_boost: float = 1.5,
)
ParameterTypeDefaultDescription
configChunkConfig | NoneNoneReads extract_keywords, keyword_max_count, compute_chunk_scores, header_score_boost from config
extract_keywordsboolTrueRun TF-IDF keyword extraction
max_keywordsint10Maximum keywords per chunk
compute_scoresboolTrueCompute importance scores
header_score_boostfloat1.5Multiplier applied to HEADER chunk scores

Public Methods

enrich(chunks: List[DocumentChunk]) → List[DocumentChunk]

Mutates and returns the same list with enriched metadata. Modifies chunk.metadata.keywords, chunk.metadata.score, and chunk.metadata.keyword_density in-place.

Parameters:

NameTypeDescription
chunksList[DocumentChunk]Chunks to enrich (modified in-place)

Returns: the same list (mutated).

Side effects: none beyond modifying the passed chunks.

Empty list: returns immediately with no work.

Keyword Extraction

TF-IDF implemented without external libraries:

code · 7 lines
1. Stopword filtering (English + Arabic stopword sets embedded in code).
2. Word tokenization: split on whitespace, strip punctuation, lowercase,
   minimum 2-character length.
3. Compute TF per chunk: word_count / total_words.
4. Compute IDF across all chunks: log(N / (1 + df)) where df = doc frequency.
5. TF-IDF score = TF × IDF.
6. Top-N by TF-IDF score → metadata.keywords (list of strings).

Single-chunk corpora get pure TF-based keywords (IDF=1 for all terms).

Scoring Algorithm

code · 17 lines
base_score = 0.0

+ keyword_density_contribution (weight 0.4):
    min(keyword_density × 10, 1.0) × 0.4

+ length_contribution (weight 0.3):
    sigmoid-like: min(char_count / 500, 1.0) × 0.3

+ position_contribution (weight 0.3):
    early chunks score higher:
    (1 - position/total_chunks) × 0.3

score = min(base_score, 1.0)

# Apply header boost:
if chunk_type == ChunkType.HEADER:
    score = min(score × header_score_boost, 1.0)

Fields Written

FieldTypeWritten When
metadata.keywordsList[str]extract_keywords=True
metadata.keyword_densityfloatextract_keywords=True
metadata.scorefloatcompute_scores=True

Examples

Standalone Enrichment

Python · 10 lines
from zerda.chunks import MetadataEnricher, StructureAwareChunker

chunker = StructureAwareChunker()
chunks = chunker.chunk(text, source="doc.md")

enricher = MetadataEnricher(max_keywords=5, header_score_boost=2.0)
enriched = enricher.enrich(chunks)

for c in enriched:
    print(c.metadata.keywords, c.metadata.score)

Disable Scoring

Python
enricher = MetadataEnricher(extract_keywords=True, compute_scores=False)

Custom Configuration

Python · 9 lines
from zerda.chunks import ChunkConfig, MetadataEnricher

config = ChunkConfig(
    extract_keywords=True,
    keyword_max_count=15,
    compute_chunk_scores=True,
    header_score_boost=2.0,
)
enricher = MetadataEnricher(config=config)

Related pages