Documents and Loaders

Models only know what they were trained on. They have never seen your notes, your PDFs, or your website.

To chat about your own files, you first need to get the text into Python.

A Document is the box LangChain puts that text in. A loader is the helper that fills the box.

The Document box

A Document has two parts. The text, and a dictionary of facts about the text.

page_content is the text. metadata says where it came from. Keep metadata, you will need it later to show sources.

Install the loaders

Most loaders live in langchain-community. Some need an extra helper package.

Load a text file

Every loader works the same. Make it, then call load(). You get a list of Documents.

A text file becomes one Document. The metadata remembers the file name.

Load a PDF

PDFs become one Document per page. The page number lands in the metadata.

Load a web page

WebBaseLoader fetches the page and strips the HTML. You get the readable text.

CSV files and whole folders

CSVLoader makes one Document per row. DirectoryLoader walks a folder and loads every matching file.

The glob pattern picks which files to load. loader_cls says which loader to use on each one.

Big folders: lazy_load

load() reads everything into memory at once. lazy_load() gives one Document at a time.

Use lazy_load when the folder is huge and you do not want to fill up your memory.

Loaders you will meet

Remember: every loader gives you a list of Documents. Text in page_content, source in metadata. The rest of the course builds on that list.