ZerdaZerda

Troubleshooting

On this page

Chunks Are Too Short

Symptom: Many chunks with char_count < 50.

Causes:

  • min_chunk_size is set too low (default 50).
  • Input has many short paragraphs or single-sentence sections.

Fix:

Python
config = ChunkConfig(min_chunk_size=150)

Or merge short chunks post-hoc:

Python · 2 lines
min_len = 100
merged = [c for c in chunks if c.char_count >= min_len]

Chunks Are Too Large

Symptom: Single chunk containing the entire document.

Causes:

  • similarity_threshold is too low (semantic mode).
  • Text has no paragraph breaks (structural mode).
  • max_chunk_size is set very high.

Fix:

Python · 8 lines
# Lower threshold → more chunks
chunker = SemanticChunker(similarity_threshold=0.85)

# Add structural splits
config = ChunkConfig(
    respect_document_structure=True,
    split_on_paragraphs=True,
)

No Keywords Extracted

Symptom: chunk.metadata.keywords == [] and chunk.metadata.score == 0.0.

Cause: Using a chunker directly without MetadataEnricher.

Fix: Use ChunkManager (runs enrichment automatically), or:

Python · 3 lines
from zerda.chunks import MetadataEnricher
enricher = MetadataEnricher()
enricher.enrich(chunks)

Arabic Text Not Split Correctly

Symptom: All Arabic text in one chunk, or splits in wrong places.

Causes:

  • Missing spacing around Arabic punctuation (common in OCR text).
  • Not using ArabicTextChunker.

Fix:

Python · 3 lines
from zerda.chunks import ArabicTextChunker

chunker = ArabicTextChunker(fix_spacing=True, chunk_size=400)

Or add normalization:

Python · 7 lines
from zerda.chunks import ArabicNormalizer, NormalizingChunkerWrapper, ArabicTextChunker

wrapper = NormalizingChunkerWrapper(
    ArabicTextChunker(fix_spacing=True),
    normalizer=ArabicNormalizer(),
)
chunks = wrapper.chunk(text)

Semantic Chunking Falls Back to TF-IDF

Symptom: You set use_semantic_chunking=True but chunks don't look semantically grouped.

Cause: sentence-transformers not installed → TF-IDF fallback activates silently.

Fix:

Bash
pip install sentence-transformers torch

ValueError: overlap >= chunk_size

Cause: ChunkConfig(overlap=512, chunk_size=256).

Fix: Ensure overlap < chunk_size.

Python
config = ChunkConfig(chunk_size=512, overlap=128)  # overlap must be < chunk_size

HTML Not Parsed Correctly

Symptom: HTML tags appearing as raw text in chunks.

Cause: beautifulsoup4 not installed; parser falls back to regex tag stripper.

Fix:

Bash
pip install beautifulsoup4

Empty List Returned

Symptom: manager.process(text) returns [].

Causes:

  • text is empty or whitespace-only.
  • All chunks shorter than min_chunk_size were dropped.

Fix:

Python · 5 lines
if not text.strip():
    print("Empty text!")
else:
    config = ChunkConfig(min_chunk_size=10)  # lower the floor
    chunks = ChunkManager(config=config).process(text)

→ FAQ →
→ Architecture →

Related pages