Large language models in production: prompting, retrieval, evals, costs and failure modes.
After fixing our retrieval (thanks for the help in my other post), now I see a different problem. In maybe 10% of the failed eval cases, the correct chunk IS in the context, I check it, but the model still answer wrong. Example: chunk say return window is 7 days for electronics and 30 days for clothes. Customer ask about headphones, bot say 30 days. Our context has 5 chunks, about 3000 tokens total, the correct one…
Three months ago our support bot go to production. Two weeks after, it tell a customer there is a discount code "SALE50". There was no such code. Customer was very angry, my boss was more angry. This is how we started to take evals seriously. Here is what I learn, maybe useful for someone. 1. Start from real conversations, not invented questions At first we write 50 test questions ourself. Bot pass almost all. Then…
Hi all, first post here. I build support chatbot for an e-commerce company, customers write in Vietnamese and English. I notice our token cost per conversation is much higher for Vietnamese. I test same FAQ answer in both language, the Vietnamese version is around 2.3x tokens of the English version with the tokenizer of our model. It make two problem: cost per conversation is higher our RAG chunk size is set by…
