Skip to content
RAG retrieve the correct chunk but model still answer wrong, why?
question1 min read67 viewsOct 16, 2026

RAG retrieve the correct chunk but model still answer wrong, why?

Nguyen Minh Anh@minhanh_ml

October 16, 2026 · 1 min read

#1 in Stochastic Parrots

67 views3 comments

After fixing our retrieval (thanks for the help in my other post), now I see a different problem. In maybe 10% of the failed eval cases, the correct chunk IS in the context, I check it, but the model still answer wrong. Example: chunk say return window is 7 days for electronics and 30 days for clothes. Customer ask about headphones, bot say 30 days.

Our context has 5 chunks, about 3000 tokens total, the correct one is usually position 3 or 4. Temperature is 0.2.

What I try:

  • "answer only from the context" in the system prompt (already have)
  • increase to top 8, it get worse

Is it the position? Or the chunk is confusing because it has 2 rules inside? How to debug this kind of problem?

Top answer

Emma Lindqvist#4 in Stochastic Parrots1 like

One trick that is not LLM-specific: ask the model to quote the sentence it is using, then answer. When it quotes the wrong sentence you immediately see whether it is a retrieval or a reading problem. You can strip the quote before the customer sees the answer.

View in the thread

#llm#rag#evals#debugging

Written by

Nguyen Minh Anh

ML engineer in Ho Chi Minh City. Building LLM support bots for e-commerce in Vietnamese and English. Evals, RAG, and explaining to boss why the bot said that.

View profile

Get the best of #stochastic-parrots weekly

One email each Monday with the most useful posts. No spam; unsubscribe in one click.

Comments

3

Sign in to join the conversation.

Sign in to comment
Lin Wei@linwei#2 in Stochastic Parrots
my guess is the chunk with 2 rules. model see "30 days" and grab it. try split the policy so each category is its own chunk, or rewrite it as a table with one category per row. also put the most relevant chunk first or last, middle position get less attention in long context
0 net votes
Emma Lindqvist@emmalindqvist#4 in Stochastic Parrots
One trick that is not LLM-specific: ask the model to quote the sentence it is using, then answer. When it quotes the wrong sentence you immediately see whether it is a retrieval or a reading problem. You can strip the quote before the customer sees the answer.
1 net votes
Nguyen Minh Anh@minhanh_ml#1 in Stochastic Parrots
Quote first is smart, I will try today. And the split by category too, I think lin is right, the chunk is confusing
1 net votes