The usual diagnosis is wrong
When an assistant over a document set answers badly, the instinct is to blame the model and reach for a better one. Usually the model performed correctly on what it was given. It was given the wrong passages.
Retrieval is a search problem sitting in front of a language problem, and it fails on its own terms. Debugging the model first means debugging the wrong component.
Look at what was retrieved before anything else
The first diagnostic is not the answer. It is the list of passages fetched for the question. Print them and read them.
Two patterns account for most failures. Either the relevant passage is simply absent, or it is present but surrounded by enough near-miss material that the salient line is buried. These have different fixes, and neither is a better model.
Chunking decides what can be found
Documents get split before indexing, and where the splits land determines what is findable. Split too small and a passage loses the context that made it meaningful. Split too large and a chunk covers several topics, so its embedding represents an average of them and matches nothing precisely.
Splitting on structure — headings, sections, natural boundaries — tends to work better than splitting on a fixed character count, because it preserves the unit the author intended. Carrying a little surrounding context into each chunk helps a passage stand on its own.
- Split on document structure, not arbitrary length
- Keep enough surrounding context that a chunk stands alone
- Store the source and location with every chunk
- Re-index when the source changes, and know how you would tell
Semantic search alone misses exact terms
Embedding similarity is good at meaning and unreliable at precision. A query containing a specific product code, error number, or proper noun can fail to retrieve the document containing that exact string, because the embedding captures general sense rather than the token.
Combining keyword matching with semantic search covers both. It is more machinery, and it removes an entire class of "why can it not find the thing I literally named".
Make the answer cite its source
Requiring each claim to point at the passage it came from does two things. It gives the reader a way to verify, and it gives you a way to debug: a wrong answer with a citation tells you immediately whether retrieval or interpretation failed.
It also constrains fabrication. A model asked to answer only from supplied passages, and to say when they do not contain the answer, produces "not found" far more often than one asked to be helpful.
Measure retrieval separately
Keep a set of real questions with the passages that should be retrieved for each. Then you can measure whether the right material was fetched, independently of what was written afterwards.
Without that separation, every change is judged on the final answer, and you cannot tell whether a prompt tweak helped or an unrelated indexing change did. Retrieval quality is measurable on its own, and measuring it is what stops the whole system being tuned by feel.
Metadata filtering removes more noise than better ranking
A large share of "the answer was wrong" turns out to be the right answer from the wrong document — a superseded version, another customer's, a different region's.
Storing structured metadata alongside each chunk and filtering before ranking removes that class entirely. Date, source, version, owner, visibility. It is less interesting than tuning similarity and it fixes more real failures, because it eliminates candidates rather than reordering them.
Decide what happens to stale content
An index is a copy, and copies go out of date. The failure is quiet: confident answers from a document that was replaced last quarter.
Decide up front how the index learns about changes, how quickly, and what happens to deleted sources. Then carry the source's date into the answer, so a reader can see they are being told something from two years ago even when the retrieval did exactly what it was asked.
Test with the questions people actually ask
Evaluation sets tend to be written by the person who built the system, which means they are phrased the way the system expects. Real questions are shorter, vaguer, full of internal shorthand, and often not questions at all.
Collect real ones as soon as there are any, and evaluate against those. The first batch is usually humbling and always more useful than the imagined set.
Know when retrieval is the wrong tool
Some questions cannot be answered by finding a passage: anything requiring a count, an aggregate, a comparison across many records, or a calculation. No amount of retrieval quality fixes "how many open tickets does this customer have", because the answer is not written down anywhere to retrieve.
Those belong to a query against structured data. Recognising which kind of question you are handling — and routing accordingly — matters more than any tuning inside the retrieval path.