Stochastic Parrots: top posts

Posts

1. Article

   Agent demo vs production: what happened when our 9-out-of-10 demo met 1,200 real tickets

   In the spring we were shown an agent demo: give it a customer request ("I was charged twice, please fix it"), and it reads the account, finds the duplicate charge, issues a refund and writes a reply. Ten scripted requests, nine handled end to end. Leadership was ready to staff down a queue. We ran a four-week shadow pilot instead: the agent processed real tickets in parallel with humans, with its actions logged but…

   Dana KowalskiAug 23, 202615likes13comments

2. Update

   Avui he descobert que el 12% de les ressenyes del nostre…

   Avui he descobert que el 12% de les ressenyes del nostre dataset d'entrenament les ha escrit un LLM. Totes comencen igual: "Com a client satisfet...". Bueno. Ara entrenem un model amb text d'un altre model. Què pot sortir malament? 🙃

   Núria PuigOct 13, 20268likes5comments

3. Article

   What I learn after 3 months building eval set for our support bot

   Three months ago our support bot go to production. Two weeks after, it tell a customer there is a discount code "SALE50". There was no such code. Customer was very angry, my boss was more angry. This is how we started to take evals seriously. Here is what I learn, maybe useful for someone. 1. Start from real conversations, not invented questions At first we write 50 test questions ourself. Bot pass almost all. Then…

   Nguyen Minh AnhOct 5, 20265likes5comments

4. Video

   For the PMs, designers and managers who keep asking me for 'something short that is not marketing': 3Blue1Brown's Large Language Models explained briefly. Short, accurate, and it does not oversell anything.

   Dana KowalskiSep 27, 20263likes4comments

5. Question

   RAG retrieve the correct chunk but model still answer wrong, why?

   After fixing our retrieval (thanks for the help in my other post), now I see a different problem. In maybe 10% of the failed eval cases, the correct chunk IS in the context, I check it, but the model still answer wrong. Example: chunk say return window is 7 days for electronics and 30 days for clothes. Customer ask about headphones, bot say 30 days. Our context has 5 chunks, about 3000 tokens total, the correct one…

   Nguyen Minh AnhOct 16, 20262likes3comments

6. Question

   Vietnamese text use 2-3x more tokens than English for same meaning, how you handle cost?

   Hi all, first post here. I build support chatbot for an e-commerce company, customers write in Vietnamese and English. I notice our token cost per conversation is much higher for Vietnamese. I test same FAQ answer in both language, the Vietnamese version is around 2.3x tokens of the English version with the tokenizer of our model. It make two problem: cost per conversation is higher our RAG chunk size is set by…

   Nguyen Minh AnhSep 16, 20262likes3comments