FAQ
On this page
General
Q: Do I need a GPU?
A: No. All rule-based modes (structural, adaptive, contextual) run on CPU with only numpy. GPU is only used by SemanticChunker and the BERT-based paths in ArabicTextChunker/MultilanguageTextChunker, and only when use_semantic_chunking=True.
Q: Which chunker should I use for RAG?
A: Start with ChunkMode.HYBRID for structured documents (Markdown, HTML) and ChunkMode.SEMANTIC for dense prose. Use ChunkMode.AUTO when unsure — it selects sensibly based on your ChunkConfig flags.
Q: How do I control chunk size?
A: Set ChunkConfig(chunk_size=512, overlap=128). Most chunkers respect these parameters. AdaptiveChunker adjusts size dynamically but uses adaptive_min_size/adaptive_max_size as bounds.
Q: Are chunks guaranteed to be ≤ chunk_size?
A: Yes, when strict_size_limit=True (default). Chunks exceeding chunk_size × (1 + size_tolerance) are hard-split. Set size_tolerance=0.0 for strict enforcement.
Arabic / Multilingual
Q: Does the library support Arabic without a GPU?
A: Yes. ArabicTextChunker(use_semantic_chunking=False) is fully rule-based and CPU-only.
Q: What language codes does MultilanguageTextChunker accept?
A: Full language names only: "arabic", "chinese", "english", etc. ISO codes like "ar" or "zh" are not supported. See the MultilanguageTextChunker docs.
Q: Should I normalize Arabic before chunking?
A: For OCR-sourced text, yes — use NormalizingChunkerWrapper with ArabicNormalizer. For clean digital text, the built-in normalization in ArabicTextChunker is usually sufficient.
Performance
Q: How do I speed up semantic chunking for large corpora?
A: Use device="cuda", increase embedding_batch_size, and set a large embedding_cache_size. Cache hits are free.
Q: Is deduplication slow?
A: Hash deduplication is O(N). Similarity deduplication is O(N²). For corpora with >10k chunks, set use_similarity=False or lower dedup_similarity_threshold to reduce comparisons.
Q: Can I run ChunkManager in parallel?
A: Yes, by creating one ChunkManager per process (not thread). The EmbeddingProvider cache is module-global and not thread-safe for concurrent writes.
Metadata
Q: When are keywords extracted?
A: After chunking, by MetadataEnricher.enrich(). Keywords are always extracted when config.extract_keywords=True (default) and you use ChunkManager. If using chunkers directly, instantiate MetadataEnricher manually.
Q: What does metadata.score represent?
A: An importance heuristic (0–1): high keyword density + medium length + early position = high score. It's useful for filtering, not for semantic ranking — use embedding similarity for that.
Errors
Q: I get ImportError: No module named 'sentence_transformers'
A: Install pip install sentence-transformers. If you don't want ML dependencies, use use_semantic_chunking=False.
Q: I get ImportError: No module named 'tiktoken'
A: Install pip install tiktoken to use TokenTextSplitter.
Q: Chunks have score=0.0 and empty keywords
A: You are using a chunker directly without MetadataEnricher. Either use ChunkManager (which runs enrichment automatically) or call MetadataEnricher().enrich(chunks) after chunking.