The most useful posts in this topic, by likes from the community.
This is the regex post I threatened in June. Context: one step of our support pipeline extracts identifiers from inbound emails: order numbers, return authorisation (RMA) codes and carrier tracking numbers. Last year, in the general enthusiasm, this step was moved from a pile of old code to an LLM call: "here is an email, return JSON with any order number, RMA code and tracking number you find". It worked. It demoed…
This topic exists because I kept seeing launches justified with a demo video and a feeling. So here is the first real post: how I evaluate an LLM feature before it ships. Nothing here is novel. That is the point. The feature, for context: classifying inbound support tickets into one of 14 categories and extracting an order ID if there is one. It is currently handled by a gradient-boosted model plus a regex (more on…
Quick context: 12 people, 5 engineers including me, TypeScript monorepo, Postgres, Vitest. In Q2 we had a mandate (from me, honestly 😅) to "use AI to fix our test debt". Coverage was 41% and our investors' tech DD person had made a face. What we did Every engineer used an AI assistant to generate tests for untested modules, about 30 minutes a day. Rule: tests must pass and get reviewed in the PR like any other…
unpopular opinion: juniors are cooked. if ur learning syntax in 2026 ur learning to ride a horse. learn to prompt, learn to ship, let the agents write the code. the devs who survive are the ones who stop typing 🚀🤖💯
Evening all. Marking on a Sunday again, so you get a post. Last term I set Year 10 (14 and 15 year olds) a short homework: explain the fetch-decode-execute cycle in your own words, 300 words, with an example. I got 28 essays back from a class of 31. At least 22 described the CPU as "the brain of the computer, orchestrating" something or other, and four mentioned a register that isn't on our spec at all. Two ended…
A short postmortem, shared with permission from the team and with details changed enough that I am comfortable posting it. Summary For 41 hours, a retrieval-augmented assistant that drafts replies for our support agents told some customers that orders could be refunded within 60 days. The policy is 30 days. Agents approve every draft before it is sent; 23 drafts with the wrong number were sent anyway. We honoured…
A ver. Three times this year a model "improved" by a lot, and three times it was leakage. The last one was a fine-tuned classifier that jumped from 0.81 to 0.93 F1, and my manager already had a slide 🙃 This is the checklist I run now before I believe anything. 1. Exact and near duplicates between train and test The 80% of my leakage problems are this. Customer reviews, support tickets, product descriptions: people…
Every few weeks somebody tells me an LLM feature will cost "basically nothing" because the per-token price is small. Then the first invoice arrives. Here is the arithmetic I now ask for before anything ships, using a real feature with the numbers rounded. (I am using illustrative prices: $3 per million input tokens and $15 per million output tokens. Plug in your provider's current numbers. The structure is the…
I keep asking people to measure, so it is only fair that I did it myself. For September I logged every coding task at work where I used an AI assistant: what it was, how long it took, how long I estimate it would have taken without, and whether I had to fix the output. The estimates are mine and biased; I tried to be conservative in both directions. The ledger tasks logged: 47 clearly faster: 29 (est. 11.5 h saved)…
Good evening. I am sorry for my poor English. I am system engineer at an SIer in Osaka and our project is the migration of a regional bank system from COBOL to Java. About 1.8 million lines of COBOL, the oldest parts written in 1987. Many people on this site and in the news say "just use AI for legacy migration". I would like to try, but I have some conditions: The bank does not allow source code to leave the…
In June our product team (four PMs including me) agreed to draft PRDs with an AI assistant for six weeks. Health tech, so no patient data went anywhere near it; we used it for structure and wording, with our own call notes as input. Some honest notes, since this topic asks for what was actually tried. What got better Blank page time went to almost zero. Getting from "notes from three customer calls" to a first draft…
weekly challenge for the AI bros: ADC reads 4095 every time the pump starts. only on site, never on bench. paste it in your chatbot. I'll wait. answer tomorrow. (hint: it is not the code)
Lo prometí en el otro post, así que aquí va en castellano (y algo de català, perdón). Teníamos un clasificador de reseñas de clientes con 5 categorías: producto, envío, precio, atención al cliente y "otro". Llevábamos dos meses probando modelos: TF-IDF con regresión logística, un BERT multilingüe afinado y un LLM con few-shot. Los tres se quedaban en 0.78-0.80 de F1. Siempre lo mismo. Sospechoso, ¿no? El experimento…
Bah, tchê. Toda semana aparece alguém no LinkedIn com "meu modelo atingiu 99% de acurácia" e um gráfico bonito. E toda semana eu tenho as mesmas três perguntas. Vou deixar aqui pra poder mandar o link 🧉 1. Qual é a proporção das classes? Se 99% das transações não são fraude, um modelo que responde "não é fraude" pra tudo tem 99% de acurácia. E não pega fraude nenhuma. Tu viu? Acurácia sozinha não diz nada quando as…
