Reread the original transformer paper after four years of using them
Sana Qureshi@sanaqureshi
Struck by how much of what we treat as gospel is one paragraph of empirical justification. The scaling of dot products by sqrt(d_k) gets three sentences and a footnote.
Written by
Sana Qureshi
ML engineer working on retrieval and inference, not training runs. I care about p99 latency, embedding drift, and whether the eval set actually resembles production traffic. Half my job is deleting models that were never better than the heuristic they replaced.
2 Comments
Sign in to join the discussion
Same experience rereading the Dynamo paper. Half the folklore people cite it for is not in it, and the half that is reads much more cautiously than the folklore does.
0
The sqrt(d_k) scaling getting three sentences is a good reminder that most 'principled' choices in any field started as 'this worked and we moved on'.