Airflow catchup ran 400 DAG runs at once and killed our warehouse, how to prevent this?
Lin Wei@linwei
Deployed new DAG yesterday with start_date=datetime(2025, 9, 1) and schedule='@daily'. Did not set catchup. Airflow scheduled ~400 runs for all past days, they start in parallel and warehouse queue full for 2 hours, other team jobs stuck. Very embarrassing.
with DAG(
dag_id='orders_daily_snapshot',
start_date=datetime(2025, 9, 1),
schedule='@daily',
) as dag:
...I paused the DAG and marked the runs failed manually. Questions:
- What is the right default so this never happen again?
- When I actually need a backfill, how to run it slowly?
Accepted answer
been there, except it was our postgres and it was 2am.
1. catchup=False on the DAG, and catchup_by_default = False under [scheduler] in airflow.cfg so new DAGs are safe even when someone forgets
2. max_active_runs=1 (or something small) on the DAG
3. put the warehouse tasks in a pool with a few slots. pools are the real safety net, they cap concurrency across ALL dags not just this one
4. when you actually want history, run it on purpose: airflow dags backfill -s 2025-09-01 -e 2025-09-30 orders_daily_snapshot, a month at a time, in quiet hours
and dont feel bad. everybody does this exactly once
Written by
Lin Wei
Data engineer at e-commerce company, Singapore. dbt, Airflow, Spark, too many dashboards. Trying to make the numbers match between finance and marketing.
Get the best of #my-model-is-fine weekly
One email each Monday with the most useful posts. No spam; unsubscribe in one click.
Comments
Sign in to join the conversation.
Sign in to commentAccepted answer
Marcus Johnson@marcus_oncall
been there, except it was our postgres and it was 2am.
1. catchup=False on the DAG, and catchup_by_default = False under [scheduler] in airflow.cfg so new DAGs are safe even when someone forgets
2. max_active_runs=1 (or something small) on the DAG
3. put the warehouse tasks in a pool with a few slots. pools are the real safety net, they cap concurrency across ALL dags not just this one
4. when you actually want history, run it on purpose: airflow dags backfill -s 2025-09-01 -e 2025-09-30 orders_daily_snapshot, a month at a time, in quiet hours
and dont feel bad. everybody does this exactly once
Siobhan O'Rourke@siobhan_tests
we have a CI check that loads every DAG and fails if catchup isn't set explicitly. five lines of pytest with DagBag. stops it at review instead of at 2am