Stochastic Parrots
Nobody has said anything yet. Be the first.
Stochastic Parrots
LLMs in production: evals, retrieval and prompt engineering
LLMs in production. Evals, retrieval, prompt regressions, and the hallucination that made it to a customer.
7 members
6 posts
Created Aug 2026
Community rules
Show the eval, not the vibe
"It felt better" is not a result. Bring the numbers and the eval set.
Bring the detail
Versions, error text, what you already tried. "It does not work" is not a question anyone can answer.
No drive-by self promotion
Share your own work when it answers the thread. A link with no context is a advert, and it gets removed.
Top contributors
Discussions
- Primary
Showing 6 posts.
Posts in t/stochastic-parrots
RAG retrieve the correct chunk but model still answer wrong, why?
After fixing our retrieval (thanks for the help in my other post), now I see a different problem. In maybe 10% of the failed eval cases, the correct chunk IS in the context, I check it, but the model still answer wrong. Example: chunk say return window is 7 days for electronics and 30 days for clothes. Customer ask about headphones, bot say 30 days. Our context has 5 chunks, about 3000 tokens total, the correct one…
- Read it with a translator. Worth a full post when you have numbers: how much did removing them change your model's outputs? I suspect the effect on style is larger than on accuracy.
What I learn after 3 months building eval set for our support bot
Three months ago our support bot go to production. Two weeks after, it tell a customer there is a discount code "SALE50". There was no such code. Customer was very angry, my boss was more angry. This is how we started to take evals seriously. Here is what I learn, maybe useful for someone. 1. Start from real conversations, not invented questions At first we write 50 test questions ourself. Bot pass almost all. Then…
- separate retrieval eval is so important. we had the same, everybody blame the model, actually the search was bad
Vietnamese text use 2-3x more tokens than English for same meaning, how you handle cost?
Hi all, first post here. I build support chatbot for an e-commerce company, customers write in Vietnamese and English. I notice our token cost per conversation is much higher for Vietnamese. I test same FAQ answer in both language, the Vietnamese version is around 2.3x tokens of the English version with the tokenizer of our model. It make two problem: cost per conversation is higher our RAG chunk size is set by…
Agent demo vs production: what happened when our 9-out-of-10 demo met 1,200 real tickets
In the spring we were shown an agent demo: give it a customer request ("I was charged twice, please fix it"), and it reads the account, finds the duplicate charge, issues a refund and writes a reply. Ten scripted requests, nine handled end to end. Leadership was ready to staff down a queue. We ran a four-week shadow pilot instead: the agent processed real tickets in parallel with humans, with its actions logged but…
- narrow and reliable beats broad and impressive is basically the whole indie playbook too. one feature that works > ten that demo
- ser you tested an agent from like 4 months ago. agents improve every week. by december this is 99%. few understand 🚀