ZerdaZerda

Deduplicator

On this page

Module: chunks.deduplicator
Import: from zerda.chunks import Deduplicator

Table of Contents

Purpose

Deduplicator removes duplicate and near-duplicate chunks from a list, using fast SHA-256 hash comparison first and optional cosine similarity comparison second.

It runs as the fourth stage in the ChunkManager pipeline, after metadata enrichment.

Constructor

Python · 7 lines
Deduplicator(
    config: Optional[ChunkConfig] = None,
    *,
    similarity_threshold: float = 0.95,
    use_hash: bool = True,
    use_similarity: bool = True,
)
ParameterTypeDefaultDescription
configChunkConfig | NoneNoneReads dedup_similarity_threshold, dedup_use_hash from config
similarity_thresholdfloat0.95Cosine similarity above which two chunks are considered duplicates
use_hashboolTrueUse SHA-256 hash comparison as a fast pre-filter
use_similarityboolTrueUse embedding-based cosine similarity comparison after hash pass

Public Methods

deduplicate(chunks: List[DocumentChunk]) → List[DocumentChunk]

Returns a new list with duplicate chunks removed. Does not mutate the input list.

Returns: List[DocumentChunk] — unique chunks in original order. First occurrence wins.

Empty input: returns [].

Single chunk: returns [chunk] (no dedup needed).

Deduplication Algorithm

code · 17 lines
Stage 1 — Hash dedup (if use_hash=True):
  seen_hashes = set()
  for each chunk:
    if chunk.content_hash in seen_hashes:
      discard
    else:
      seen_hashes.add(chunk.content_hash)
      keep chunk

Stage 2 — Similarity dedup (if use_similarity=True):
  embed all surviving chunk texts via EmbeddingProvider
  for each new chunk i:
    compute cosine similarity against all previously kept chunks
    if max(similarity) >= similarity_threshold:
      discard (near-duplicate)
    else:
      keep

content_hash is computed in DocumentChunk.__post_init__ as sha256(text.encode()).hexdigest().

Hash-Based vs Similarity-Based

ModeSpeedCatches
Hash onlyO(N)Exact duplicates
Similarity onlyO(N²)Near-duplicates
Both (default)O(N) + O(M²) where M < NExact + near-duplicates

Use use_similarity=False for large corpora (>10k chunks) where similarity is too slow.

Complexity

ConfigurationTimeSpace
Hash onlyO(N)O(N)
Similarity onlyO(N² × D)O(N × D)
BothO(N) + O(M² × D)O(N × D)

Where N = total chunks, M = hash-unique chunks, D = embedding dimension.

Examples

Standalone Use

Python · 9 lines
from zerda.chunks import Deduplicator, SemanticChunker

chunker = SemanticChunker()
chunks = chunker.chunk(text_with_duplicates)

dedup = Deduplicator(similarity_threshold=0.95)
unique = dedup.deduplicate(chunks)

print(f"Before: {len(chunks)}, After: {len(unique)}")

Hash Only (Fast Mode)

Python · 2 lines
dedup = Deduplicator(use_similarity=False)
unique = dedup.deduplicate(chunks)

Custom Threshold

Python · 2 lines
# More aggressive: remove chunks that are 85%+ similar
dedup = Deduplicator(similarity_threshold=0.85)

Disable in ChunkManager

Python · 2 lines
config = ChunkConfig(deduplication_enabled=False)
manager = ChunkManager(config=config)

Related pages