Pandera, Great Expectations or plain asserts for data validation in a small team?
Núria Puig@nuria_data
Hola! Small data team (3 people), Python, pandas, some dbt, Airflow. We have zero data validation, and last month a supplier changed a column from euros to cents without telling anybody. Our revenue dashboard showed 100x for two days 🙃 Nobody noticed until the CFO was very happy, and then very angry.
I want to add validation at the ingestion step. The options I see:
- pandera: schemas in Python, looks light.
- Great Expectations: very complete, but last time I tried it was a lot of setup and config.
- dbt tests: we already use dbt for some models.
- Plain asserts in the pipeline. Bueno, the honest option.
What do you use in a small team, and what actually catches the euros-to-cents kind of problem? Gràcies!
Accepted answer
Small team, I'd do two layers.
1. pandera at ingestion for schema and ranges. It's just Python, it reads like documentation and it fails loudly:
import pandera as pa
schema = pa.DataFrameSchema({
'amount_eur': pa.Column(float, pa.Check.in_range(0, 50_000)),
'currency': pa.Column(str, pa.Check.isin(['EUR'])),
})
schema.validate(df, lazy=True)
2. dbt tests (not_null, unique, accepted_values, plus a custom one) on the models you already have.
For euros to cents specifically, a range check won't catch it if the max is generous. What catches it is a distribution check against history: today's median amount vs the last 30 days. If it moves 10x, fail. That's a 15-line custom test in dbt. GX can do this too, but for 3 people the setup cost is real.
Get the best of #my-model-is-fine weekly
One email each Monday with the most useful posts. No spam; unsubscribe in one click.
Comments
Sign in to join the conversation.
Sign in to commentAccepted answer
Lin Wei@linwei
Small team, I'd do two layers.
1. pandera at ingestion for schema and ranges. It's just Python, it reads like documentation and it fails loudly:
import pandera as pa
schema = pa.DataFrameSchema({
'amount_eur': pa.Column(float, pa.Check.in_range(0, 50_000)),
'currency': pa.Column(str, pa.Check.isin(['EUR'])),
})
schema.validate(df, lazy=True)
2. dbt tests (not_null, unique, accepted_values, plus a custom one) on the models you already have.
For euros to cents specifically, a range check won't catch it if the max is generous. What catches it is a distribution check against history: today's median amount vs the last 30 days. If it moves 10x, fail. That's a 15-line custom test in dbt. GX can do this too, but for 3 people the setup cost is real.
Núria Puig@nuria_data
the distribution check against history is exactly what I was missing. a range check would pass, the cents were still 'reasonable' numbers. gràcies!!
Emma Lindqvist@emmalindqvist
Same pattern for energy meter data: we compare each day total with the same weekday over the last four weeks and alert if it falls outside a band. It has caught kWh vs Wh twice. Unit changes are the most common silent failure I know.
Mo Siddiqui@mo_boston
plain asserts are underrated tbh. one assert that the median amount is under 1000 at the top of the script caught more stuff at my job than our fancy tool lol. start there, move to pandera when it gets annoying
Dana Kowalski@dana_ml
Agree with lin's two layers. One addition: make the failure message say what to do. 'Median amount 412.00 vs 30-day median 4.12: possible unit change, check the supplier feed' saves the 2am person ten minutes.
Aoife Brennan@aoifeb_dev
We went with GX at my last job, team of 4, and regretted it. Too much config for what we needed. pandera plus dbt tests is the sweet spot for small teams imo.