Text Splitters

You loaded a 300 page PDF. Now what? You cannot send all of it to the model at once.

A text splitter cuts long text into small pieces called chunks.

Think of cutting a book into pages. Each page is small enough to read alone, and you can find the right page fast.

Why chunks?

Install

The one splitter you need most

RecursiveCharacterTextSplitter tries to cut at paragraphs first, then sentences, then words. So chunks stay readable.

chunk_size is the biggest a piece can be, in characters. chunk_overlap is how much each piece repeats from the one before.

split_text takes a string and gives a list of strings.

Why overlap?

Imagine a sentence cut right in the middle. Half is in one chunk, half in the next. Both halves lose their meaning.

Overlap repeats a little text at each cut, so no idea gets sliced in two. Around 10 percent of the chunk size is a good start.

Splitting Documents, not just strings

Usually you have Documents from a loader. Use split_documents. It keeps the metadata on every chunk.

Every chunk remembers which file and page it came from. Later you can show the user where an answer was found.

The simple splitter

CharacterTextSplitter cuts on one separator only. It is less clever, but easy to predict.

It cuts only at blank lines. Good when your text already has clean paragraphs.

Splitting code

Code has its own shape. Functions and classes should stay together. LangChain knows the rules for many languages.

from_language picks separators like def and class, so a chunk usually holds a whole function.

Picking a chunk size

Tip: There is no perfect chunk size. Try a few, ask real questions, and keep the size that finds the best answers.