How do I get a model to answer from my own documents?

Ask a language model what your company's refund policy is and it will tell you. Fluently, in your tone, with the structure of a real policy. It will also be invented, because the model has never seen your refund policy and has no way to say so.

This is the first thing to understand about building on models: not knowing and being wrong look identical from the outside. The sentence arrives with the same confidence either way.

The two ways out, and why one of them is a trap

There are only two things you can do about a model that does not know your material.

Teach it. Fine-tuning adjusts the model's weights on your data, so the knowledge lives inside it. This is the intuitive answer, and for facts it is usually the wrong one. It is slow and costs money every time your documents change. It cannot tell you where an answer came from. And it does not reliably stop invention — a fine-tuned model is still a model, still fluent, still willing.

Show it. Retrieval finds the passages relevant to a question and puts them in front of the model at the moment you ask, with an instruction to answer from those and nothing else. The knowledge stays in your documents. Change a document and the next answer changes. Ask where a claim came from and the system can point at the passage.

The rule of thumb: fine-tuning changes how a model behaves, retrieval changes what it knows. Tone, format and style are behaviour. Your refund policy is not.

What retrieval actually does

Four steps, and none of them are clever.

Split the documents. A whole document is too big to hand over, so it is cut into chunks of a few hundred words. This sounds like an implementation detail and is the decision most likely to quietly ruin your results.

Turn each chunk into numbers. An embedding model reads a chunk and produces a long list of numbers — a few hundred of them — positioned so that passages about similar things end up with similar lists. That is the whole trick: meaning, converted into something a computer can measure distance between.

Store them. In a database that can search by closeness rather than by keyword.

At question time, embed the question the same way, find the chunks nearest to it, and hand those to the model with the question and one instruction: answer from these passages, and say so if they do not contain the answer.

That last clause is doing more work than everything before it.

Why this is worth the trouble

It can cite. Because the answer was generated from specific passages, the system knows which ones. A reader who does not trust an answer can check it — and a system that can be checked is a different product from one that cannot.

It stays current. Update the document, re-embed that chunk, done. No retraining, no waiting.

It can decline. A model told to answer only from what it was given can report that it was not given enough. That is impossible when the knowledge is baked into weights, because there is nothing to be missing from.

Where it goes wrong

Retrieval does not fix invention. It makes it traceable, which is a different and more useful thing.

The failure that matters is quiet: the search returns the wrong passages, the model answers from them faithfully, and you get a confident answer built on irrelevant material. No error, no warning. The pipeline did its job; it was handed the wrong input.

Which is why the parts of this that look like plumbing — how big the chunks are, what counts as close enough, whether you can measure any of it — are where the work actually is. The four steps above take an afternoon. Making them right on your material is the subject of everything else here.

Start with the chunks. They decide what one embedding has to speak for, and that turns out to be the whole game.