Chunks — AI-Powered Intelligent Chunking System

Production-Grade RAG Chunking Library for Arabic, multilingual, and English text.

What is Chunks?

Chunks is a production-ready Python library that splits documents into optimally-sized, semantically coherent pieces (chunks) for use in Retrieval-Augmented Generation (RAG) pipelines, vector databases, and LLM applications.

It provides:

• Six specialized chunking strategies (semantic, adaptive, structural, contextual, Arabic, multilingual)
• A unified orchestrator (ChunkManager) that selects the right strategy automatically
• Rich metadata on every chunk (keywords, importance score, position, character offsets)
• Built-in deduplication (hash-based and similarity-based)
• First-class Arabic NLP support with normalization pipeline
• LangChain-compatible text splitters
• Zero mandatory dependencies for rule-based mode; optional sentence-transformers, torch, transformers, tiktoken, beautifulsoup4 for advanced features

Quick Example

from zerda.chunks import SemanticChunker, ChunkConfig

config = ChunkConfig(chunk_size=512, overlap=128)
chunker = SemanticChunker(config=config)

text = "Your document text goes here..."
chunks = chunker.chunk(text, doc_id="",source="my_doc.pdf")

Navigation

Class Reference

Design Philosophy

1. Correctness first — Every chunk preserves meaning. No mid-sentence cuts.
2. Strategy diversity — Different documents need different strategies; the library exposes all of them.
3. Arabic-first multilingual — Arabic is a first-class language, not an afterthought.
4. Composable pipeline — Each component can be used standalone or orchestrated by ChunkManager.
5. Graceful degradation — Heavy dependencies (torch, transformers) are optional; the library falls back to rule-based modes automatically.