Your model didn't get better, your test set leaked (a checklist)

A ver. Three times this year a model "improved" by a lot, and three times it was leakage. The last one was a fine-tuned classifier that jumped from 0.81 to 0.93 F1, and my manager already had a slide 🙃 This is the checklist I run now before I believe anything.

1. Exact and near duplicates between train and test

The 80% of my leakage problems are this. Customer reviews, support tickets, product descriptions: people copy paste, templates repeat. Exact duplicates are easy:

import pandas as pd

def norm(s: pd.Series) -> pd.Series:
    return (s.str.lower()
             .str.replace(r"\s+", " ", regex=True)
             .str.replace(r"[^\w ]", "", regex=True)
             .str.strip())

train["key"] = norm(train["text"])
test["key"] = norm(test["text"])
leaked = test["key"].isin(set(train["key"]))
print(f"{leaked.mean():.1%} of test rows also in train")

In my case: 11.4% of the test set. Eleven! The "improvement" was mostly the model remembering those.

For near duplicates (the same text with one word changed, or a template with a different name) I use MinHash from the datasketch library, Jaccard threshold around 0.8 on word 3-grams. Slower, but it finds the templates.

from datasketch import MinHash, MinHashLSH

def mh(text, k=3, num_perm=128):
    words = text.split()
    m = MinHash(num_perm=num_perm)
    for i in range(max(len(words) - k + 1, 1)):
        m.update(" ".join(words[i:i + k]).encode("utf8"))
    return m

lsh = MinHashLSH(threshold=0.8, num_perm=128)
for i, t in train["key"].items():
    lsh.insert(f"tr{i}", mh(t))

near = [i for i, t in test["key"].items() if lsh.query(mh(t))]
print(len(near), "test rows have a near-duplicate in train")

2. Split by the thing that repeats, not by row

If one customer writes 40 tickets and 30 go to train and 10 to test, the model learns that customer's style. Use a group split:

from sklearn.model_selection import GroupShuffleSplit

gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(gss.split(df, groups=df["customer_id"]))

Same for time. If your data has dates and production will see the future, split by date. A random split on time series is lying to yourself, doncs.

3. Features that know the answer

The classic: a column "resolution_time" in a model that predicts whether a ticket will be escalated. Escalated tickets take longer to resolve. The model is 0.95 AUC and useless, because at prediction time you don't know the resolution time. Ask it for every feature: do I have this value at the moment I make the prediction?

4. For LLMs: was the benchmark in the training data?

With LLMs we cannot check the training data, so I do the opposite: I keep a small private test set that was never on the internet. 150 examples written by our team from real cases, never pasted into any chat tool. If a model is great on the public benchmark and only normal on our 150, I believe our 150.

5. The sanity check nobody does: shuffle the labels

Train on shuffled labels. Performance should drop to chance. If it doesn't, something in the pipeline is leaking (an ID column, the row order, a feature built with the full dataset). It took me 10 minutes and I found a target-encoded feature computed before the split. Vale.

After fixing all of this, the fine-tuned classifier was 0.84. Still better than 0.81! Just a smaller slide.