What is Chunking?
Chunking is the process of breaking large datasets, documents, or files into smaller, fixed-size or semantically meaningful segments. It is a common data preprocessing step in AI/ML pipelines to manage memory and enable efficient processing.
Chunking splits data according to rules such as byte limits, token counts, sentence boundaries, or topic shifts. The resulting pieces are easier to store, index, and feed into models that have context-length restrictions.
In practice, chunking often includes overlap between adjacent chunks to preserve context across boundaries. Overlap size and chunk size are tunable hyperparameters that trade off between completeness and computational cost.
The technique supports parallel processing, streaming, and retrieval operations because each chunk can be handled independently by different workers or stored in vector databases.
Example
A 100-page PDF report is split into 500-word chunks with 50-word overlaps so each chunk can be embedded and stored separately for a retrieval system.
Why it matters
Modern AI systems routinely handle data volumes far larger than model context windows or single-machine memory, making chunking essential for scalable training, fine-tuning, and retrieval-augmented generation.
Frequently asked questions
Tokenization breaks text into individual tokens or subwords, while chunking groups those tokens into larger segments for processing or storage.
Related terms
Tokenization is the process of breaking text into smaller units called tokens that language models can process numerically.
An embedding (or vector embedding) is a way to represent words, sentences, or other data as dense numerical vectors in a high-dimensional space so that similar items end up close together.
A vector database is a specialized database designed to store and query high-dimensional vector embeddings, enabling fast similarity searches instead of traditional exact-match queries.
Batch size is the number of training examples processed together in a single forward and backward pass during model training.
Cosine similarity measures how similar two vectors are by computing the cosine of the angle between them, ignoring their magnitudes.
Data augmentation is a technique that artificially increases the size and diversity of a training dataset by creating modified versions of existing data samples.