Here is the plain version first. A company runs on data that moves. Orders come in, payments clear, support tickets pile up. None of that data is useful where it lands. It has to be picked up, cleaned, checked, and put somewhere the dashboards and the reports can read it. That moving and cleaning is a data pipeline. DataOps is the discipline of running those pipelines the way I run software: versioned, tested, watched, and owned. It exists because a pipeline nobody watches breaks quietly. The dashboard still loads. The numbers are just wrong. I have seen a team make a decision off a dashboard that was three days stale, and nobody knew, because the pipeline had failed on a Tuesday and no alarm fired. This lesson follows one nightly pipeline run, nine hops, from raw data landing to a dashboard a human can trust.
Every hop charges rent in minutes, not milliseconds. The bar below is a worked example for this lesson's run, not a measurement from a live system, but the shape is honest: in a healthy run the transform is the longest hop, and that is fine, because it is the hop that does the real work. Switch the scenario and watch the bar change. In the renamed column scenario the contract and quality hops grow, because the run stops, gets fixed, and re-runs. In the backfill scenario the transform and publish hops grow, because a month of history is a month of work.
The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the copy, the transform, the tests, and the total run time move. The numbers are a teaching example, not live measurements. The shape is the lesson: the rows and the history decide the run time, and the contract decides whether the extra time is spent catching a problem early or cleaning it up late.
How many million rows arrive each day. Each million rows adds about 2 minutes to the transform, because the joins and sums have more rows to chew through.
Share of columns covered by quality tests. Each tested column costs time, because a test reads the data it checks.
How many past days this run re-processes. Each backfill day re-runs a day of transforms, so the transform grows with it.
On means the contract check runs before the transform and catches bad data early. Off means the check is skipped, the quality tests catch the problem later, and the publish is delayed while someone fixes it.
Inside the nightly window. The dashboard is fresh before the morning standup, and nobody had to wait.
Try this: (1) Leave the controls at the defaults: 10 million rows a day, 60 percent test coverage, no backfill, contract on. Transform about 25 minutes, quality about 14 minutes, total about 72 minutes. (2) Turn the contract off and change nothing else. The total goes to about 79 minutes, and the extra 7 minutes is quality tests and publish cleaning up a problem the contract would have caught in 5. (3) Now push rows to 50 million, coverage to 100 percent, and backfill to 30 days. Transform about 405 minutes, quality about 102 minutes, total about 575 minutes. Same pipeline, same team. Only the size and the history changed.
Three questions. No score is saved anywhere — this is just you against the material.
The same topic four times, from the first plain idea to the level where I run it in production and debug it at 2 AM. Each level adds one thing the level below cannot do.
Definition: DataOps is the discipline of running data pipelines the way I run software: versioned, tested, watched, and owned. A company runs on data that moves. Orders come in, payments clear, support tickets pile up. None of that data is useful where it lands. It has to be picked up, cleaned, checked, and put somewhere the dashboards and the reports can read it. That moving and cleaning is a data pipeline.
Types:
Uses: Use it the moment a dashboard drives a decision. The nightly orders pipeline is the example from this lesson. A pipeline nobody watches breaks quietly: the dashboard still loads, the numbers are just wrong. I have seen a team make a decision off a dashboard that was three days stale, and nobody knew, because the pipeline had failed on a Tuesday and no alarm fired.
Dimensions: At low volume one run a night is fine. At 10x the rows the transform gets slow and splits by day and region. What breaks is a failed run with no freshness alarm: a silent failure, and the dashboard lies by omission. It costs storage for raw and transformed data, compute for the transform, and time to write the tests. You give up moving fast on untested pipelines for pipelines you can trust at 2 AM.
Definition: A data contract is a promise between the app team and the pipeline, with a version. When the app team adds a new column, that is version 1.1, and old readers ignore it. When they rename a column, that is version 2.0, because old readers break, and a breaking change needs a date, a notice, and a migration window. I never accept a breaking change with no notice. The rule I use is simple: additive changes are fine any time, breaking changes go through a version bump and a two week notice.
Types:
Uses: Use it when the app team renames a column. Renaming is a breaking change: old readers break, so it needs a date, a notice, and a migration window. The contract file carries the version, and the pipeline checks the version before it checks the columns.
Dimensions: At 100x the rows the raw zone becomes the bottleneck, so I partition it by day and by source, and the quality tests run on samples instead of the full data, with a full run once a week. What breaks is a currency bug doubling the average order value overnight: the distribution alarm catches it, and the freshness and volume alarms do not. That is why I alert on all three. It costs the contract, the registry, and the test runs on every change. You give up changing schemas freely for a version bump and a notice. The app team moves slower and the pipeline stops breaking.
Definition: Idempotent is a plain word with a plain meaning: running the same job twice gives the same result as running it once. I get there by writing each day's data into its own partition and replacing the whole partition on a re-run, never appending to it. The backfill deletes the day, then writes the day. If it crashes halfway, I run it again and the result is the same.
Types:
Uses: Use it at 2 AM when the first backfill run failed halfway. A backfill that appends doubles the data the second time it runs, and the second run is always the one that happens at 2 AM when the first one failed. A backfill that replaces is a fix. A backfill that appends is a second incident.
Dimensions: At 10x the rows the transform splits by day and region and runs in parallel, because a transform that runs in one thread is a transform that waits. What breaks is a schema the pipeline does not know it is reading: a pipeline that does not know which schema version it is reading is a pipeline that guesses. The safe changes are adding a column, adding a new value to a list, and widening a number. The dangerous ones are renaming a column, deleting a column, and changing a type. It costs partitions, re-runs, and storage for both raw and transformed data. You give up the simplicity of appending for the safety of replace. Storage is cheap. Doubled data is not.
Definition: At this level the pipeline is a product with an owner, a version, tests in CI, observability on freshness, volume, and distribution, and a runbook. DataOps is not ETL with a new name. ETL is the pattern: extract, transform, load. DataOps is the discipline around the pattern: version control for the pipeline code, tests for the data, CI that runs the tests on every change, observability that watches freshness and volume, and an owner for every pipeline. ETL tells you how the data moves. DataOps tells you how you know the move worked.
Types:
Uses: Use it when the dashboard drives real decisions. Ten million orders a day, a freshness promise of 120 minutes, and a test that fails the run on one duplicate order, one negative amount, or one missing customer. A team that does ETL without DataOps has a pipeline. A team that does DataOps has a pipeline they can trust at 2 AM.
Dimensions: At 100x the discipline is the same and only the plumbing scales: sampled tests, partitioned storage, parallel transforms. What breaks is the failure no single signal catches, which is why all three signals get alerts. A freshness alarm fires when the data is late. A volume alarm fires when the row count jumps or crashes. A distribution alarm fires when the average order value doubles overnight, which usually means a currency bug, not a sales miracle. It costs the pipeline, the tests, the storage, and the on-call. You give up ad-hoc data movement entirely. Every change goes through review, tests, and a version bump, or it does not ship.
The lesson above is one nightly run through nine hops. Production is the rest of it. Here is what I actually spend my time on once the pipeline is running and the dashboard is fresh.
A data contract is a promise, and like any promise it needs a version. When the app team adds a new column, that is version 1.1, and old readers ignore it. When they rename a column, that is version 2.0, because old readers break, and a breaking change needs a date, a notice, and a migration window. I never accept a breaking change with no notice. The rule I use is simple: additive changes are fine any time, breaking changes go through a version bump and a two week notice. The contract file carries the version, and the pipeline checks the version before it checks the columns.
Schema evolution is the polite name for the same problem. Data changes shape over time, and the pipeline has to survive it. The safe changes are adding a column, adding a new value to a list, and widening a number. The dangerous changes are renaming a column, deleting a column, and changing a type. I handle the safe ones automatically and the dangerous ones by hand, with a version bump. The schema registry is where the versions live, and the pipeline reads it at the start of every run. A pipeline that does not know which schema version it is reading is a pipeline that guesses.
Idempotent backfills are the difference between a fix and a second incident. Idempotent is a plain word with a plain meaning: running the same job twice gives the same result as running it once. I get there by writing each day's data into its own partition and replacing the whole partition on a re-run, never appending to it. The backfill deletes the day, then writes the day. If it crashes halfway, I run it again and the result is the same. A backfill that appends is a backfill that doubles the data the second time it runs, and the second run is always the one that happens at 2 AM when the first one failed.
DataOps is not ETL with a new name. ETL is the pattern: extract, transform, load. DataOps is the discipline around the pattern: version control for the pipeline code, tests for the data, CI that runs the tests on every change, observability that watches freshness and volume, and an owner for every pipeline. ETL tells you how the data moves. DataOps tells you how you know the move worked. A team that does ETL without DataOps has a pipeline. A team that does DataOps has a pipeline they can trust at 2 AM.
Observability for data has three signals, and I watch all three. Freshness is how old the newest data is. Volume is how many rows arrived. Distribution is what the data looks like: the average, the spread, the share of empty values. A freshness alarm fires when the data is late. A volume alarm fires when the row count jumps or crashes. A distribution alarm fires when the average order value doubles overnight, which usually means a currency bug, not a sales miracle. I alert on all three, because each one catches a failure the other two miss.
At 10x the rows, the shape holds and the transform gets slow. I split the transform by day and by region and run the pieces in parallel, because a transform that runs in one thread is a transform that waits. At 100x, the raw zone becomes the bottleneck, so I partition it by day and by source, and the quality tests run on samples instead of the full data, with a full run once a week. The contract does not change. The tests do not change. The only thing that scales is the discipline, and the discipline is what this lesson is.
This is the config shape I would actually write: the contract for the orders dataset, with its version, owner, and freshness promise, and the quality test that runs after the transform.
# data contract: orders, version 2.0
# agreed with the app team; breaking changes need a version bump and two weeks' notice
dataset: orders
version: "2.0"
owner: app-team
sla:
freshness_minutes: 120
expected_rows_per_day: 10000000
schema:
- column: order_id
type: string
required: true
unique: true
- column: customer_id
type: string
required: true
- column: amount_usd
type: decimal
required: true
rule: "amount_usd >= 0"
- column: region
type: string
required: true
allowed_values: [us-east, us-west, eu-west, ap-south]
# quality test: run after the transform, before the publish
# -- no duplicate orders, no negative amounts, no missing customers
# SELECT COUNT(*) FROM transformed_orders
# WHERE order_id IN (SELECT order_id FROM transformed_orders
# GROUP BY order_id HAVING COUNT(*) > 1)
# OR amount_usd < 0 OR customer_id IS NULL;
# -- a result above zero fails the run
What the snippet means: the contract names the dataset, its version, its owner, and the freshness promise of 120 minutes. The schema lists every column, its type, and whether it is required. The amount has a rule, zero or above, and the region has a short list of allowed values. The test underneath is the check that runs after the transform. It counts duplicate order ids, negative amounts, and missing customer ids. Any count above zero fails the run before the bad data reaches the warehouse.
This one stayed with me because the pipeline said success for three nights while the data was wrong, and nobody knew until finance asked why revenue was down. Treat this as an illustrative scenario, a composite of a pattern I have seen, not one company's incident report.
A nightly pipeline fed the revenue dashboard for a small online store. A bug in the transform dropped orders from one region for three days. The pipeline log said success every night, because the transform did not fail. It just produced fewer rows. Nobody noticed for three days, because the dashboard still loaded and the numbers looked plausible. On the fourth day the finance team asked why revenue was down, and the answer was that it was not down. The data was.
Three things. First, a volume check on every run: if the row count for any region drops by more than 10 percent from the day before, the run fails and pages me. A quiet drop is now a loud failure. Second, a quality test that counts rows by region and compares with yesterday, so the check lives in the pipeline, not in my head. Third, the freshness and volume alarms go to the on-call phone, not to an email folder nobody reads. A pipeline that says success while the data is wrong is worse than a pipeline that fails, because a failure gets fixed and a false success gets believed.
This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real pipeline. Take a minute with it before you read my approach.
Design the daily data pipeline for an online store that takes 10 million order rows a day and must have the revenue dashboard fresh by 7 AM. The data comes from the orders service, the payments service, and the customer service. The dashboard must never show a half loaded day, and a bad day of data must never reach the finance team.
I would ask five things before I sized anything. What time does the data land, and what happens if one source is late: does the run wait, or does it run without it? How fresh is fresh enough: is 7 AM a hard deadline, and who gets paged if the run misses it? What does bad data look like here: duplicates, negative amounts, missing customers, or all three? Who owns each source, and who do I call at 2 AM when a column changes? And what is the recovery plan: if today is wrong, do we fix it in place or backfill the day?
Start with the numbers from this lesson. At 10 million rows a day and a transform of about 2 minutes per million rows, the transform is about 25 minutes including the fixed cost. Quality tests at 60 percent coverage add about 14 minutes. Raw copy, contract, lineage, publish, watching, and notifying add about 33 minutes. The whole run is about 72 minutes. If the data lands at midnight, the dashboard is fresh by about 1:15 AM, well inside the 7 AM deadline, with more than five hours of headroom. Storage: 10 million rows a day at roughly 200 bytes a row is about 2 GB a day in the raw zone, about 60 GB a month, about 730 GB a year. Keep a year of raw data. It is cheap, and it is the only way back. Three sources means three contracts, one per team, each with an owner and a freshness promise.
Waiting for a late source buys completeness and costs freshness: a run that waits for a source that never arrives misses the deadline. I wait until 1 AM, then run with what I have and mark the dashboard as partial. Batch once a day buys simplicity and costs freshness: the dashboard is a day old by design, and if the business needs hourly numbers, this design is wrong. Testing every column buys safety and costs time: 100 percent coverage at 10 million rows is about 22 minutes of tests, and I would rather spend that time than explain a wrong dashboard to finance. One warehouse for everyone buys simplicity and costs isolation: a bad publish touches every dashboard at once, so I publish in one swap and keep the old version as a fallback.
Saved in this browser, so the Lessons index can show your progress across the course.