Primary
AI Hype Check
LLMs, agents, copilots: what works, what is marketing
Using AI tools for real work and saying honestly how it went. Coding assistants, agents, local models, evals, costs, the jobs question, and the demo that did not survive contact with production. Hype and doom both get checked.
20 members
16 posts
Created Jun 2026
+1
Community rules
Say what you actually tried
Model, tool, task, result. "AI will replace us all" with nothing behind it gets removed.
Label AI written text
If a model wrote your post, say so.
No drive-by self promotion
Your own stuff is fine when it answers the thread. A bare link with no context is an advert.
Moderated by Dana Kowalski
Top contributors
Discussions
What works marketing evals costs
Real use cases, measured gains, honest limits
Showing 16 posts.
Posts in t/ai_hype_check
- "a coin is cheaper" is going in my quotes file
- For every junior reading this: I interview 10+ people a month. Every single round still has 'explain this code' and 'debug this'. Fundamentals are the filter. Ignore this tweet, keep studying.
- Okay da, prompt your way through this: STM32 motor controller resets only when the pump starts, only in field, never on the bench. Watchdog not firing. Go. I wait.
I logged every AI-assisted coding task for a month. Here is the honest ledger
I keep asking people to measure, so it is only fair that I did it myself. For September I logged every coding task at work where I used an AI assistant: what it was, how long it took, how long I estimate it would have taken without, and whether I had to fix the output. The estimates are mine and biased; I tried to be conservative in both directions. The ledger tasks logged: 47 clearly faster: 29 (est. 11.5 h saved)…
- The paired-samples point is the one I'd underline. Every 'juniors are cooked' take assumes the tool replaces judgement. In your log it amplifies judgement and punishes the absence of it.
- local models on my homelab GPU for autocomplete, works ok for bash and docker compose. not as smart but nothing leaves my network
- voted none machane, copilot is company money. my own money goes to biriyani
We replaced an LLM feature with a regex and nobody noticed
This is the regex post I threatened in June. Context: one step of our support pipeline extracts identifiers from inbound emails: order numbers, return authorisation (RMA) codes and carrier tracking numbers. Last year, in the general enthusiasm, this step was moved from a pile of old code to an LLM call: "here is an email, return JSON with any order number, RMA code and tracking number you find". It worked. It demoed…
- imagine flexing a regex in 2026 ngmi. agents will write and maintain the regex AND the LLM. devs are just middlemen fren
- I don't write regex but I understood this whole post, and that's a skill. Also 'attributed to the servers being upgraded' is the funniest line I've read all month 😂
We let AI write our tests for a quarter. Coverage went up, confidence did not
Quick context: 12 people, 5 engineers including me, TypeScript monorepo, Postgres, Vitest. In Q2 we had a mandate (from me, honestly 😅) to "use AI to fix our test debt". Coverage was 41% and our investors' tech DD person had made a face. What we did Every engineer used an AI assistant to generate tests for untested modules, about 30 minutes a day. Rule: tests must pass and get reviewed in the PR like any other…
- 'The test enshrines the bug' is the sentence. I've been trying to explain this to my director for months. Coverage measures execution, not verification. Mutation score is the honest number, so glad you tracked it.
Year 10 handed me 28 nearly identical AI essays on how a CPU works. Here's what I do now
Evening all. Marking on a Sunday again, so you get a post. Last term I set Year 10 (14 and 15 year olds) a short homework: explain the fetch-decode-execute cycle in your own words, 300 words, with an example. I got 28 essays back from a class of 31. At least 22 described the CPU as "the brain of the computer, orchestrating" something or other, and four mentioned a register that isn't on our spec at all. Two ended…
- Parsons problems are so underrated! I use them for my JS bootcamp, plus 'predict before you run'. The students who argue with me about their prediction learn the most.
Teu modelo tem 99% de acurácia? Bah, então me mostra a matriz de confusão
Bah, tchê. Toda semana aparece alguém no LinkedIn com "meu modelo atingiu 99% de acurácia" e um gráfico bonito. E toda semana eu tenho as mesmas três perguntas. Vou deixar aqui pra poder mandar o link 🧉 1. Qual é a proporção das classes? Se 99% das transações não são fraude, um modelo que responde "não é fraude" pra tudo tem 99% de acurácia. E não pega fraude nenhuma. Tu viu? Acurácia sozinha não diz nada quando as…
- olha a cientista de dados vindo humilhar os backend kkkkkk. mas sério, a parte da planilha do gerente perdendo só 4 pontos é ouro, véi. mostra isso pro PO e vê a cara dele
- vist! I send it to everyone at work who asks me "what is a token". It works better than my explanations, una mica trist ahah
Antes de culpar al modelo, mira tus etiquetas
Lo prometí en el otro post, así que aquí va en castellano (y algo de català, perdón). Teníamos un clasificador de reseñas de clientes con 5 categorías: producto, envío, precio, atención al cliente y "otro". Llevábamos dos meses probando modelos: TF-IDF con regresión logística, un BERT multilingüe afinado y un LLM con few-shot. Los tres se quedaban en 0.78-0.80 de F1. Siempre lo mismo. Sospechoso, ¿no? El experimento…
- Vaya tela con la empresa externa. Déjame adivinar: la contratasteis por precio y etiquetaban becarios a destajo. Clásico de cárnica. Buen post, Núria, aunque seas del Barça.
The cost-per-request maths nobody does before launch
Every few weeks somebody tells me an LLM feature will cost "basically nothing" because the per-token price is small. Then the first invoice arrives. Here is the arithmetic I now ask for before anything ships, using a real feature with the numbers rounded. (I am using illustrative prices: $3 per million input tokens and $15 per million output tokens. Plug in your provider's current numbers. The structure is the…
- $52k a month to write emails. aye right. could hire a room of folk in Glasgow for that and they'd make the tea too
Our PMs started drafting PRDs with AI. Six weeks later, this is what I would keep
In June our product team (four PMs including me) agreed to draft PRDs with an AI assistant for six weeks. Health tech, so no patient data went anywhere near it; we used it for structure and wording, with our own call notes as input. Some honest notes, since this topic asks for what was actually tried. What got better Blank page time went to almost zero. Getting from "notes from three customer calls" to a first draft…
- Late, but sharing this with my PM group. The 'beautiful problem statement based on a single call' failure mode is the one I'd tattoo on every new PM. Writing the problem by hand is such a simple rule. Stealing it.
If I may ask: how do you use AI for COBOL to Java migration when the bank forbids internet?
Good evening. I am sorry for my poor English. I am system engineer at an SIer in Osaka and our project is the migration of a regional bank system from COBOL to Java. About 1.8 million lines of COBOL, the oldest parts written in 1987. Many people on this site and in the news say "just use AI for legacy migration". I would like to try, but I have some conditions: The bank does not allow source code to leave the…
Accepted answer
Walt here, 30-odd years on mainframes. Honest answer: AI is useful for the part nobody wants to do, and useless for the part that matters. Useful: explaining a single paragraph of COBOL in plain language, drafting Java for a self-contained calculation, writing test cases from a copybook, summarising what a JCL job does. A local model can do that fine. I tried a small one on an air-gapped box last year. Not useful: knowing why the interest rounding on line 40,212 is different from the one on line 12,007. That is a business rule from 1994 that somebody's manager decided on a phone call. It is not in the comments and it is not in your Excel. The only way to find it is parallel runs: old system and new system on the same input, compare every output field, every night, for months. If you take one thing: build the comparison harness first. The AI can help you write it, ironically.- Kazuhiro, I did a smaller version of this in 2004, RPG to Java for an insurer. Walt is right. The code was the easy part; the undocumented behaviour the business depended on was the project. We found one batch job that existed only to correct the output of another batch job. Nobody alive knew why. Be patient, keep a list of every surprise, and never let anyone switch the old system off early. Cheers, G.
Your model didn't get better, your test set leaked (a checklist)
A ver. Three times this year a model "improved" by a lot, and three times it was leakage. The last one was a fine-tuned classifier that jumped from 0.81 to 0.93 F1, and my manager already had a slide 🙃 This is the checklist I run now before I believe anything. 1. Exact and near duplicates between train and test The 80% of my leakage problems are this. Customer reviews, support tickets, product descriptions: people…
- no soy de datos pero lo entendí todo, gracias Núria 💜 ¿lo vas a traducir para codigo_en_espanol?
Postmortem: our support assistant invented a refund policy
A short postmortem, shared with permission from the team and with details changed enough that I am comfortable posting it. Summary For 41 hours, a retrieval-augmented assistant that drafts replies for our support agents told some customers that orders could be refunded within 60 days. The policy is 30 days. Agents approve every draft before it is sent; 23 drafts with the wrong number were sent anyway. We honoured…
- 'the model was the least broken part of the system' should be printed on every AI incident report. it's also the shape of 90% of security incidents : a stale document, a confident system, and a human who stopped reading.
How I build evals for an LLM feature (the boring version)
This topic exists because I kept seeing launches justified with a demo video and a feeling. So here is the first real post: how I evaluate an LLM feature before it ships. Nothing here is novel. That is the point. The feature, for context: classifying inbound support tickets into one of 14 categories and extracting an order ID if there is one. It is currently handled by a gradient-boosted model plus a regex (more on…
- ok i feel personally attacked by 'you know that it demos'. saving this. the 30 person-hours for the golden set is the part i'd never budget for and probably the part i most need
- The Spanish and Portuguese slice is the bit nobody does. Every vendor demo I have seen is in American English. Our users write Swahili, Sheng and English in the same message and that is exactly where things fall apart.