ZerdaZerda

Text Splitters

On this page

Module: chunks.text_splitter
Import: from zerda.chunks import CharacterTextSplitter, RecursiveCharacterTextSplitter, TokenTextSplitter, SentenceTextSplitter

LangChain-compatible text splitters. All inherit from TextSplitter (see Base Classes).

Table of Contents

TextSplitter Base

All splitters share these methods:

Python · 8 lines
class TextSplitter(ABC):
    def split_text(self, text: str) -> List[str]: ...           # abstract
    def split_documents(self, documents: List[Document]) -> List[Document]: ...
    def create_documents(
        self,
        texts: List[str],
        metadatas: Optional[List[dict]] = None,
    ) -> List[Document]: ...
  • split_documents splits each Document.page_content and returns new Document objects with preserved (and extended) metadata.
  • create_documents wraps plain strings in Document objects then splits.

CharacterTextSplitter

Splits text at a single separator string, then merges pieces back up to chunk_size with chunk_overlap characters of context.

Python · 9 lines
CharacterTextSplitter(
    separator: str = "\n\n",
    chunk_size: int = 1000,
    chunk_overlap: int = 200,
    length_function: Callable[[str], int] = len,
    keep_separator: bool = False,
    strip_whitespace: bool = True,
    config: Optional[ChunkConfig] = None,
)

Parameters

ParameterTypeDefaultDescription
separatorstr"\n\n"String to split on
chunk_sizeint1000Target characters per chunk
chunk_overlapint200Overlap in characters
length_functionCallablelenFunction to measure chunk length
keep_separatorboolFalseInclude separator in output pieces
strip_whitespaceboolTrueStrip leading/trailing whitespace
configChunkConfig | NoneNoneReads chunk_size and overlap from config

Algorithm

  1. Split text on separator.
  2. Filter empty strings.
  3. Merge adjacent pieces into chunks ≤ chunk_size, adding chunk_overlap context between.
  4. Return List[str].

Example

Python · 4 lines
from zerda.chunks import CharacterTextSplitter

splitter = CharacterTextSplitter(separator="\n\n", chunk_size=500, chunk_overlap=50)
texts = splitter.split_text(raw_text)

RecursiveCharacterTextSplitter

Tries a list of separators from most-to-least preferred, recursively splitting until all pieces are ≤ chunk_size.

Python · 9 lines
RecursiveCharacterTextSplitter(
    separators: Optional[List[str]] = None,
    chunk_size: int = 1000,
    chunk_overlap: int = 200,
    length_function: Callable[[str], int] = len,
    keep_separator: bool = False,
    arabic_mode: bool = False,
    config: Optional[ChunkConfig] = None,
)

Parameters

ParameterTypeDefaultDescription
separatorsList[str] | NoneNoneSeparator hierarchy. See defaults below.
chunk_sizeint1000Target characters per chunk
chunk_overlapint200Overlap in characters
length_functionCallablelenLength measurement function
keep_separatorboolFalseInclude separator in output
arabic_modeboolFalseUse Arabic separator hierarchy
configChunkConfig | NoneNoneReads chunk_size and overlap from config

Default Separators

English (arabic_mode=False):

Python
["\n\n", "\n", ". ", "! ", "? ", "; ", ", ", " ", ""]

Arabic (arabic_mode=True):

Python
["\n\n", "\n", "؟ ", "! ", ". ", "؛ ", "، ", " ", ""]

Algorithm

code · 7 lines
split_text(text):
  for sep in separators:
    splits = text.split(sep)
    good = [s for s in splits if len(s) <= chunk_size]
    oversized = [s for s in splits if len(s) > chunk_size]
    recursively split oversized with remaining separators
  merge good pieces up to chunk_size with overlap

Examples

Python · 13 lines
from zerda.chunks import RecursiveCharacterTextSplitter

# English
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
texts = splitter.split_text(document)

# Arabic
arabic_splitter = RecursiveCharacterTextSplitter(
    arabic_mode=True,
    chunk_size=400,
    chunk_overlap=80,
)
texts = arabic_splitter.split_text(arabic_document)

TokenTextSplitter

Splits text by token count using tiktoken. Requires pip install tiktoken.

Python · 6 lines
TokenTextSplitter(
    encoding_name: str = "cl100k_base",
    chunk_size: int = 1000,
    chunk_overlap: int = 200,
    config: Optional[ChunkConfig] = None,
)

Parameters

ParameterTypeDefaultDescription
encoding_namestr"cl100k_base"tiktoken encoding. Also accepts "gpt2", "p50k_base", etc.
chunk_sizeint1000Target tokens per chunk
chunk_overlapint200Overlap in tokens
configChunkConfig | NoneNoneReads model_token as encoding name, chunk_size, overlap

Algorithm

  1. tiktoken.get_encoding(encoding_name) — loads encoding (cached).
  2. encoding.encode(text) → token ID list.
  3. Slides a window of chunk_size tokens with chunk_overlap step.
  4. encoding.decode(window) → string chunk.

Raises

  • ImportError if tiktoken is not installed.

Example

Python · 8 lines
from zerda.chunks import TokenTextSplitter

splitter = TokenTextSplitter(
    encoding_name="cl100k_base",
    chunk_size=256,
    chunk_overlap=32,
)
texts = splitter.split_text(long_text)

SentenceTextSplitter

Splits at sentence boundaries (., !, ?, ؟), then groups sentences into chunks by character count.

Python · 6 lines
SentenceTextSplitter(
    chunk_size: int = 1000,
    chunk_overlap: int = 200,
    sentence_endings: Optional[List[str]] = None,
    config: Optional[ChunkConfig] = None,
)

Parameters

ParameterTypeDefaultDescription
chunk_sizeint1000Target characters per chunk
chunk_overlapint200Overlap in characters
sentence_endingsList[str] | NoneNoneOverride sentence end tokens. Default: [".", "!", "?", "؟"]
configChunkConfig | NoneNoneReads chunk_size and overlap

Algorithm

  1. Regex split on [.!?؟]+\s+.
  2. Group sentences into chunks ≤ chunk_size.
  3. Apply character overlap by carrying tail sentences.

Example

Python · 4 lines
from zerda.chunks import SentenceTextSplitter

splitter = SentenceTextSplitter(chunk_size=400, chunk_overlap=60)
texts = splitter.split_text(article_text)

Common Patterns

LangChain-Style Document Splitting

All splitters implement split_documents():

Python · 6 lines
from zerda.chunks import RecursiveCharacterTextSplitter, Document

docs = [Document(page_content=t, metadata={"source": f}) for t, f in pairs]
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
split_docs = splitter.split_documents(docs)
# Returns List[Document] with inherited + chunk_index metadata

Creating Documents from Strings

Python · 3 lines
texts = ["chunk one", "chunk two"]
metadatas = [{"page": 1}, {"page": 2}]
docs = splitter.create_documents(texts, metadatas=metadatas)

Choosing a Splitter

Use CaseRecommended Splitter
General markdown/proseRecursiveCharacterTextSplitter
Token-budget-aware (GPT-4 context)TokenTextSplitter
Arabic textRecursiveCharacterTextSplitter(arabic_mode=True)
Sentence-coherent chunksSentenceTextSplitter
Simple separator splitCharacterTextSplitter

Related pages