Posts tagged #data-engineering on Dynamo Geeks: 5 posts from the community, most useful first.
Deployed new DAG yesterday with start_date=datetime(2025, 9, 1) and schedule='@daily' . Did not set catchup. Airflow scheduled ~400 runs for all past days, they start in parallel and warehouse queue full for 2 hours, other team jobs stuck. Very embarrassing. with DAG( dag_id='orders_daily_snapshot', start_date=datetime(2025, 9, 1), schedule='@daily', ) as dag: ... I paused the DAG and marked the runs failed…
Every month same meeting. Marketing dashboard say revenue X, finance report say Y, difference 4-8%, and data team (me) is asked to explain. After 6 months I made a checklist. Sharing in case it help someone. Where the numbers split Gross vs net. Marketing count order value at checkout. Finance count after cancellations, refunds and returns. Biggest gap is always this one. Which date. Order date, payment date,…
If you move money, at some point finance will ask "does our ledger match the bank". The honest answer at most startups is "mostly". This post is how we went from "mostly" to a daily report that finance actually trusts. It took us around 4 months, so dont feel bad if yours is still a spreadsheet. Three sources, not two For card and UPI payments you have three views of the same money: Your ledger: what you think…
We are getting more charger and meter data than our current setup was designed for. Right now it is about 40,000 devices sending a reading every 15 seconds, so roughly 230 million rows per day, into a single Postgres table with a B-tree index on (device_id, ts) . Inserts are still fine, but queries over a week for one site are getting slow and the table is huge. Typical queries: last 24 hours for one device, at full…
Using dbt on BigQuery. Orders model is incremental, source table get updated when order status change (paid, shipped, refunded). Model look like this: {{ config(materialized='incremental') }} select order_id, status, amount, updated_at from {{ source('shop', 'orders') }} {% if is_incremental() %} where updated_at > (select max(updated_at) from {{ this }}) {% endif %} Problem 1: same order_id appear multiple times…
