I logged every AI-assisted coding task for a month. Here is the honest ledger
Dana Kowalski@dana_ml
Contents
I keep asking people to measure, so it is only fair that I did it myself. For September I logged every coding task at work where I used an AI assistant: what it was, how long it took, how long I estimate it would have taken without, and whether I had to fix the output. The estimates are mine and biased; I tried to be conservative in both directions.
The ledger
tasks logged: 47
clearly faster: 29 (est. 11.5 h saved)
about the same: 11
slower (fixing confident errors): 7 (est. 4.0 h lost)
net: about 7.5 h saved
time spent coding that month: about 70 h
Roughly a 10% gain on coding time. Real, useful, and nowhere near 10x.
Where it saved time
- Boilerplate: argument parsing, plotting code, test fixtures, SQL for a schema I already understood. The biggest wins and the least interesting.
- Unfamiliar APIs: "how do I do X in library Y", where I could check the answer immediately by running it.
- Translation: a pandas snippet to SQL, a bash loop to Python.
Where it cost time
- Seven tasks where the answer looked right and was not: a deprecated parameter, an off-by-one in a date window, and a statistical test that assumed independence when my samples were paired. That last one cost the most, because the code ran fine and the p-value looked lovely.
- The pattern: it costs time when I cannot verify the output faster than I could have written it.
What I would actually conclude
- The gain is concentrated in tasks where checking is cheap. Choose tasks accordingly.
- Seniority matters in a direction people do not expect: the paired-samples error would have gone unnoticed by someone who did not already know the answer. The tool makes me faster partly because I can tell when it is wrong.
- I will keep using it. I will not claim it doubled my output, because it did not.
Jo, I believe you owe me a spreadsheet.
Get the best of #ai_hype_check weekly
One email each Monday with the most useful posts. No spam; unsubscribe in one click.
Comments
Sign in to join the conversation.
Sign in to commentJo Brennan@jo_ships
ok ok. my october log is running. preliminary numbers after 2 weeks: i think my '10x' is more like 2x on the boring stuff and 0.5x on anything involving stripe webhooks. you were right and i'm annoyed about it 🐶
Dana Kowalski@dana_ml
2x on the boring stuff is a genuinely good result for a solo founder, because you have a lot of boring stuff. I look forward to the spreadsheet.
Kenji Tanabe@kenjitanabe
The paired-samples point is the one I'd underline. Every 'juniors are cooked' take assumes the tool replaces judgement. In your log it amplifies judgement and punishes the absence of it.
Ryan Goh@ai_maxi_ryan
10% because ur using it wrong. agents doing whole features end to end is where the 10x is. ur still in autocomplete land
Dana Kowalski@dana_ml
Possibly. I would love to see your log. Same format: task, time, estimate without, fixes needed. One month.
Mio Kobayashi@mio_rustacean
Our team did a similar log for two weeks. Similar result, around 10 to 15%. The biggest variance was by person, not by tool.
Renee Whitfield@renee_em
Sharing this with my managers. When leadership asks 'how much faster are we with AI', this is what an honest answer looks like. Would you mind if I used your table format for a team survey?