One day of events is 100 GB of JSON lines in an object store. It is also a model that flags a risky session, and a reason the flag can be trusted. Between the raw day and the trusted flag there is a pipeline with nine hops, and each hop can lie to the next one. This lesson walks the day through ingest, a Spark batch job, a Flink stream, a warehouse, MLflow tracking, a Kubeflow training run, and a SHAP explanation. I built the story around the failure I watch for most: one hot key that makes the whole job wait on one worker, and one schema change that makes the pipeline quietly wrong.
Difficulty: Advanced · Know this first: Lesson 122, Lesson 80, Lesson 09
By the end of this lesson you will have run a whole data pipeline on your own machine, at small scale, and you will have the numbers to prove each hop works. First, a local Spark job that cleans 200 event rows and counts errors per session. Second, the salted-key fix, with the largest partition printed before and after. Third, an MLflow run that logs a model, its data version, and its score. Fourth, a SHAP explanation of one flagged session, with the reasons ranked. Fifth, a lineage manifest that hashes the data and the code, the way DVC does. You will also know where Spark ends and Flink begins, and why a model without its data version is not reproducible.
What you need: Python 3.12 and a Java runtime on your own machine. Spark runs in local mode here, and local mode needs a JVM. No cluster, no cloud account, no API keys. The Flink job and the Kubeflow pipeline are reader-side: the lab shows the Flink window logic as a small local script and the Kubeflow pipeline as a file you syntax-check, because neither a stream cluster nor a Kubeflow cluster lives in a lesson. The outputs shown are the outputs that ran on this machine, and they are marked that way.
Every data pipeline I have worked on is the same shape with different names. Data arrives. Something checks it. Something cleans it. Something stores it. Something learns from it. Something explains what it learned. The names change: Kafka, Apache Spark, Apache Flink, a warehouse, MLflow, Kubeflow, SHAP. The shape does not. Learn the shape once, and every pipeline you meet is a variation on it.
The day in this lesson is 100 GB of app events: clicks, errors, logins. About 20 million rows. The goal is a flag: which sessions look risky, and why. Here is the trip, hop by hop, with the number I watch at each one.
| Hop | What it does | The number I watch |
|---|---|---|
| Source events | The app writes JSON lines to the object store | File count and total bytes. A missing hour is a missing hour. |
| Ingest | Checks each row against the schema, quarantines the bad ones | Dead-letter queue depth. Growth means the schema changed. |
| Spark transform | Cleans the day in parallel, joins session to user | Largest partition. One hot key sets the job time. |
| Flink stream | Counts errors live in five-minute windows | Watermark. A stuck watermark means the stream is stuck. |
| Warehouse write | Stores the clean day, partitioned by day and type | Row count after the write. The gap must match the dedup and quarantine counts. |
| MLflow tracking | Logs parameters, data version, metrics, model file | Run id. A score with no run id is a story, not a result. |
| Kubeflow training | Runs load, split, train, score, register as steps | Which step failed. A failed split and a failed train are two different bugs. |
| SHAP explain | Says what each feature added to one flag | Top feature. If one feature explains every flag, check for a leak. |
| Decision | A human reads the reasons and decides | Verdict written. A flag with no verdict gets read twice. |
Ingestion is the first design decision, and it has three honest answers. Batch files land on a schedule and get processed on a schedule. A stream arrives as it happens and gets processed as it happens. Change data capture reads the database log and ships every change as an event. Batch is simple and late. A stream is fast and harder to run. Change data capture is exact and tied to one database. This lesson uses the first two: files for the day, a stream for the last five minutes.
| Question | Spark (batch) | Flink (stream) |
|---|---|---|
| What it answers | What happened today | What is happening now |
| How it reads | Files, whole or in slices | Events, one at a time, in windows |
| When it is late | By the schedule: hourly, nightly | By the watermark: seconds, if the stream is healthy |
| What breaks it | A hot key in the shuffle | A stuck watermark, or late events |
| What it costs | Cluster time per run | A job that never stops |
Three tools record three different pasts, and a team needs all three. DVC versions the data: the hash of the exact bytes a model learned from. MLflow versions the run: the parameters, the score, and the model file. Kubeflow versions the pipeline: the steps, in order, so the run can be repeated. A model with a run id but no data version can be repeated on the wrong data. A model with a data version but no run id cannot be repeated at all.
Write down, in one line each, what DVC records, what MLflow records, and what Kubeflow records. Then write down which one you would open first if the model score dropped after a retrain and the code did not change. The answer is MLflow, and the run id is the first thing you read. If you opened the pipeline YAML first, you were looking for a code bug in a run whose code did not change. Start with the run.
The diagram above follows one day of events through the pipeline. These four paths are the same idea drawn four ways. One day, cleaned in batch. One day, broken by a schema change. One day, counted row by row. One day, answered live. Run each path. One run takes 7.2 seconds across the path, so you can watch each hop light up in turn.
The return rule: the green dot only starts moving after the orange dot reaches the data hop. A return with no request is a lie. The diagram never draws one. The failure replay at the end of Run all four shows the other side of the rule. Break the Spark job, and the orange dot stops at Spark and never comes back.
One day of events walks the batch pipeline from the object store to the flag. Ingest lands it, Spark cleans it, the warehouse keeps it, MLflow logs the run, Kubeflow trains it, SHAP explains it, and the decision goes back to the team.
Not run yet.
Same day, one difference: a new field lands with no schema update. Ingest quarantines 400,000 rows, Spark retries on a day with a hole in it, and Flink crashes on the new field. The dot stops at Flink and never comes back. The warehouse never gets the live count.
Not run yet.
The same day, counted row by row instead of timed hop by hop. Twenty million rows go in. Forty thousand bad rows are quarantined at ingest, and Spark drops 560,000 duplicates. Nineteen point four million clean rows reach the warehouse, and the training job samples 500,000 of them. The drop is 600,000 rows, and every one of them is accounted for.
Not run yet.
The same day, answered live instead of in batch. Ingest lands the stream, Flink counts errors in a five-minute window, the warehouse keeps the count, and the decision goes back before the batch job even starts. Training is skipped. The live trip does not wait for it.
Not run yet.
Every hop charges rent in milliseconds. The bar below is a worked example for one day of events through the pipeline, not a measurement from a live system, but the shape is honest. The training hop is the big one. It costs 600 ms at the simulator defaults, and the Spark hop costs 320 ms. The other hops are plumbing. Switch the scenario and watch the bar change. In the hot key story the Spark hop grows to 1,400 ms, because one worker does the work of six. In the schema drift story two hops disappear, because the stream crashed and the training never ran. Move a simulator slider and watch the Spark hop and the training hop move with it.
The budget bar above shows three fixed stories. This simulator lets you write your own. It answers the trade-off at the heart of this lesson: more data makes every hop slower, more partitions spread the work thinner, and the hot key punishes the partition count you chose. You set how much data lands per day, how many Spark partitions split it, how much of it arrives live, and how many rows SHAP explains. Move a slider and watch the Spark hop, the training hop, and the budget total change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: the slowest partition sets the time for the whole job.
How many gigabytes of events land per day. One hundred GB is about 20 million rows. Five hundred GB is about 100 million rows.
How many slices Spark cuts the day into. More partitions means thinner slices and a faster job, until the slices get so small the bookkeeping costs more than the work.
How much of the day arrives as a live stream instead of a batch file. The live share is the Flink hop. The rest waits for the batch job.
How many flagged rows SHAP explains. Every row costs about a millisecond. Explaining every flag is dear. Explaining a sample is the compromise.
Try this: (1) At the defaults, 100 GB a day, 16 partitions, 30 percent live, 100 rows explained, the Spark hop is 320 ms and the training hop is 600 ms. The day is 20,000,000 events, and the budget total is 1,360 ms. (2) Drop to 10 GB, 4 partitions, 10 percent live, 10 rows explained. The Spark hop falls to 152 ms and the training hop to 240 ms. The budget total is 585 ms. Four partitions is enough for 10 GB, and the small day is cheap. (3) Push to 500 GB, 4 partitions, 80 percent live, 500 rows explained. The Spark hop grows to 5,640 ms, because 500 GB cut into 4 slices is 125 GB a slice. The training hop grows to 2,200 ms. The budget total is 9,300 ms. From the defaults, the big day with too few partitions costs 7,940 ms more per slice. That is the trade, priced. Now raise the partitions to 64 and watch the Spark hop fall.
Formulas, in plain terms. Events per day = gigabytes times 200,000. Ingest hop = 10 ms plus 0.3 ms per gigabyte. Spark hop = 40 ms plus gigabytes times 2.8 ms times 16 divided by partitions. Flink hop = 30 ms plus 2 ms per live percent. Warehouse hop = 20 ms plus 1 ms per gigabyte. MLflow hop = 25 ms, fixed. Training hop = 200 ms plus 4 ms per gigabyte. SHAP hop = 50 ms plus 1 ms per explained row. Decision hop = 15 ms, fixed. The budget total is the nine hops added up. The timings are a teaching example, not live measurements. The shape will not change when the numbers do.
Do this now, on your own machine. You will run the Spark job, fix the skew with a salted key, log an MLflow run, explain one flag with SHAP, and write the lineage manifest. The Python parts run on this machine and their outputs are shown exactly as they ran. The Flink job and the Kubeflow pipeline are reader-side, and the lab says so at each step. Your run ids and hashes will differ. The shapes will not.
Save this file as spark_local.py. It builds 200 event rows in PySpark local mode, drops the 12 bad ones, and counts errors per session. Spark local mode needs a Java runtime. If Java is missing, the job fails before it starts, and the error says so plainly. The script also turns the Python daemon off, because the daemon fork path fails in sandboxed VMs. On a normal machine you can leave it on.
"""spark_local.py: the local Spark job for Lesson 183.
PySpark in local mode. It builds 200 event rows, drops the bad ones,
and counts errors per session. Run it with the venv python:
./de-venv/bin/python spark_local.py
"""
import os
import random
import sys
# Spark launches its Python workers with PYSPARK_PYTHON. Point it at
# this same interpreter, or the worker daemon dies on import.
os.environ.setdefault("PYSPARK_PYTHON", sys.executable)
os.environ.setdefault("PYSPARK_DRIVER_PYTHON", sys.executable)
from pyspark.sql import SparkSession
from pyspark.sql.functions import col
random.seed(183)
KINDS = ["click", "click", "click", "error", "login"]
rows = []
for i in range(200):
session = "s-%03d" % (i % 20)
rows.append((session, random.choice(KINDS), 1700000000 + i))
# 12 bad rows: 8 with no session, 4 with an unknown event kind.
for i in range(8):
rows.append((None, "click", 1700000000 + i))
for i in range(4):
rows.append(("s-001", "bogus", 1700000000 + i))
spark = (SparkSession.builder.master("local[2]").appName("lesson-183")
# The daemon fork path fails in sandboxed VMs, so workers are
# spawned directly. On a normal machine you can leave it on.
.config("spark.python.use.daemon", "false")
.getOrCreate())
spark.sparkContext.setLogLevel("ERROR")
df = spark.createDataFrame(rows, ["session", "event", "ts"])
print("rows in: %d" % df.count())
clean = df.filter(col("session").isNotNull() & col("event").isin("click", "error", "login"))
print("rows clean: %d" % clean.count())
print("bad rows dropped: %d" % (df.count() - clean.count()))
errors = clean.filter(col("event") == "error").groupBy("session").count()
print("sessions: %d" % clean.select("session").distinct().count())
print("sessions with errors: %d" % errors.count())
for row in errors.orderBy(col("count").desc()).limit(5).collect():
print("session %s: %d errors" % (row["session"], row["count"]))
spark.stop()
./de-venv/bin/python spark_local.py
Expected output, exactly as it ran on this machine:
rows in: 212 rows clean: 200 bad rows dropped: 12 sessions: 20 sessions with errors: 19 session s-009: 3 errors session s-012: 3 errors session s-019: 3 errors session s-006: 3 errors session s-018: 3 errors
Read the three numbers first. Two hundred rows in, 188 clean, 12 bad dropped. Then read the sessions with errors. The job is small on purpose. The shape is the shape of the real job: read, clean, group, count.
Save this file as spark_skew.py. One hot session holds 80 of 200 rows. Grouping by the raw session id puts all 80 in one partition. Salting the key spreads it, and the script prints the largest partition before and after.
"""spark_skew.py: the salted-key fix for Lesson 183.
One hot session holds 80 of 200 rows. Grouping by the raw session id
puts all 80 in one partition, and the job waits on that worker. Salting
the key spreads the hot session across partitions, and the largest
partition shrinks. Run it with the venv python:
./de-venv/bin/python spark_skew.py
"""
import os
import random
import sys
# Spark launches its Python workers with PYSPARK_PYTHON. Point it at
# this same interpreter, or the worker daemon dies on import.
os.environ.setdefault("PYSPARK_PYTHON", sys.executable)
os.environ.setdefault("PYSPARK_DRIVER_PYTHON", sys.executable)
from pyspark.sql import SparkSession
random.seed(183)
rows = []
for i in range(80):
rows.append(("s-hot", "error"))
for i in range(120):
rows.append(("s-%03d" % (i % 12), "click"))
spark = (SparkSession.builder.master("local[2]").appName("lesson-183-skew")
# The daemon fork path fails in sandboxed VMs, so workers are
# spawned directly. On a normal machine you can leave it on.
.config("spark.python.use.daemon", "false")
.getOrCreate())
spark.sparkContext.setLogLevel("ERROR")
sc = spark.sparkContext
def largest_partition(rdd):
return max(rdd.glom().map(len).collect())
by_key = sc.parallelize(rows).map(lambda r: (r[0], r[1])).partitionBy(4)
print("largest partition before salting: %d rows" % largest_partition(by_key))
salted = sc.parallelize(rows).map(
lambda r: ((r[0], random.randint(0, 7)), r[1])).partitionBy(4)
print("largest partition after salting: %d rows" % largest_partition(salted))
spark.stop()
./de-venv/bin/python spark_skew.py
Expected output, exactly as it ran on this machine:
largest partition before salting: 90 rows largest partition after salting: 61 rows
Now change the salt range from 8 to 2 and run it again. The largest partition grows back. The salt is the fix, and the range is the dose. Change it back when you are done.
Save this file as mlflow_local.py. It trains a tiny logistic regression on synthetic session features and logs the run to a local SQLite store, mlflow.db: the model, the data version, the row count, the accuracy, and the model file. MLflow 3 no longer accepts the file store, so the local store is SQLite.
"""mlflow_local.py: local MLflow tracking for Lesson 183.
Trains a tiny logistic regression on synthetic session features and
logs the run to a local SQLite store (mlflow.db): parameters, the data
version, the accuracy, and the model file. MLflow 3 no longer accepts
the file store, so the local store is SQLite. Run it with the venv
python: ./de-venv/bin/python mlflow_local.py
"""
import mlflow
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
rng = np.random.default_rng(183)
n = 400
error_count = rng.integers(0, 8, size=n)
clicks = rng.integers(5, 120, size=n)
session_minutes = rng.integers(1, 90, size=n)
night = rng.integers(0, 2, size=n)
X = np.column_stack([error_count, clicks, session_minutes, night])
risky = ((error_count >= 4) | ((error_count >= 2) & (night == 1))).astype(int)
y = risky
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=183)
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
score = accuracy_score(y_test, model.predict(X_test))
mlflow.set_tracking_uri("sqlite:///mlflow.db")
mlflow.set_experiment("risky-session")
with mlflow.start_run(run_name="risky-session-v3") as run:
mlflow.log_param("model", "logistic-regression")
mlflow.log_param("data_version", "events-v3")
mlflow.log_param("rows", n)
mlflow.log_metric("accuracy", score)
mlflow.sklearn.log_model(model, name="model")
print("run id: %s" % run.info.run_id)
print("model: logistic-regression")
print("data version: events-v3")
print("rows: %d" % n)
print("accuracy: %.4f" % score)
./de-venv/bin/python mlflow_local.py
Expected output, exactly as it ran on this machine:
run id: fa5ca1e733eb4cac9cdfcb05af66d252 model: logistic-regression data version: events-v3 rows: 400 accuracy: 0.9800
Your run id will differ. The data version will not: events-v3. Open the mlflow.db store with the MLflow UI and find the run by its id. A score with no run id is a story. This run has one.
Save this file as shap_local.py. It trains the same tiny model and explains one flagged session: error count 6, 40 clicks, 55 minutes, at night. SHAP says what each feature added to the score.
"""shap_local.py: SHAP on a small model for Lesson 183.
Trains the same tiny logistic regression as mlflow_local.py and
explains one flagged session with SHAP. The top features are the
reasons the flag carries. Run it with the venv python:
./de-venv/bin/python shap_local.py
"""
import numpy as np
import shap
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
FEATURES = ["error_count", "clicks", "session_minutes", "night"]
rng = np.random.default_rng(183)
n = 400
error_count = rng.integers(0, 8, size=n)
clicks = rng.integers(5, 120, size=n)
session_minutes = rng.integers(1, 90, size=n)
night = rng.integers(0, 2, size=n)
X = np.column_stack([error_count, clicks, session_minutes, night])
y = ((error_count >= 4) | ((error_count >= 2) & (night == 1))).astype(int)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=183)
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
row = np.array([[6, 40, 55, 1]])
print("flagged session: error_count=6, clicks=40, session_minutes=55, night=1")
print("model score: %.4f" % model.predict_proba(row)[0][1])
explainer = shap.Explainer(model.predict, X_train)
values = explainer(row)
ranked = sorted(zip(FEATURES, values.values[0]),
key=lambda pair: abs(pair[1]), reverse=True)
print("SHAP values for the flagged session:")
for name, value in ranked:
print(" %s: %+.4f" % (name, value))
./de-venv/bin/python shap_local.py
Expected output, exactly as it ran on this machine:
flagged session: error_count=6, clicks=40, session_minutes=55, night=1 model score: 1.0000 SHAP values for the flagged session: error_count: +0.3167 night: +0.0633 session_minutes: +0.0117 clicks: -0.0017
Read the top feature first. Error count carries the flag, and night makes it worse. If one feature explained every flag, that is the moment to check for a leak, not the moment to celebrate the accuracy.
Save this file as lineage.py. DVC records the hash of the data file and the code that cleaned it. This script does the same bookkeeping by hand with the standard library: it writes a small events file, hashes the data and the cleaning code, and writes lineage.json.
"""lineage.py: the DVC stand-in for Lesson 183.
DVC records the hash of the data file and the code that cleaned it, so a
retrain can name the exact bytes it learned from. This script does the
same bookkeeping by hand, with the standard library only: it writes a
small deterministic events file, hashes the data and the cleaning code,
and writes lineage.json. Run it from the work dir.
"""
import hashlib
import json
import random
from pathlib import Path
HERE = Path(__file__).resolve().parent
def sha256_file(path):
h = hashlib.sha256()
h.update(Path(path).read_bytes())
return h.hexdigest()
def main():
random.seed(183)
rows = []
for i in range(200):
session = "s-%03d" % (i % 20)
kind = random.choice(["click", "click", "click", "error", "login"])
rows.append("%s,%s,%s" % (session, kind, 1700000000 + i))
data_path = HERE / "events_sample.csv"
data_path.write_text("session,event,ts\n" + "\n".join(rows) + "\n", encoding="utf-8")
code_path = HERE / "spark_local.py"
manifest = {
"data_file": data_path.name,
"data_rows": len(rows),
"data_sha256": sha256_file(data_path),
"code_file": code_path.name,
"code_sha256": sha256_file(code_path),
"model_version": "risky-session-v3",
}
out = HERE / "lineage.json"
out.write_text(json.dumps(manifest, indent=2) + "\n", encoding="utf-8")
print("data rows: %d" % manifest["data_rows"])
print("data sha256: %s" % manifest["data_sha256"])
print("code sha256: %s" % manifest["code_sha256"])
print("model version: %s" % manifest["model_version"])
print("wrote %s" % out.name)
if __name__ == "__main__":
main()
python3 lineage.py
Expected output, exactly as it ran on this machine:
data rows: 200 data sha256: ffd7adf72797d2c44c4bdabb75a7f0d0234e6a68f4975e15c99a7ebc90d30af1 code sha256: 1de4c5aef9d47f2388f54191bdbe7d836a0dff4f4ba887eaaa8956e900a43b73 model version: risky-session-v3 wrote lineage.json
The DVC stage file that says the same thing in DVC words is in Go deeper. It was validated by parsing it, and its two stages are clean and train. The Kubeflow pipeline file, kfp_pipeline.py, was syntax-checked with ast.parse on this machine. There is no Kubeflow cluster in this VM, so the pipeline is reader-side: the step names and the order are the design, and the cluster is yours to bring.
There is no stream cluster in this VM, so the Flink job is yours to run where Flink lives. The logic is five lines, and it is the whole of hop 4: keep a five-minute window, count errors per session, flag a session at three errors, and watch the watermark. A watermark that stops moving means the stream is stuck, and every count after it is a guess. The window is the idea. The cluster is the plumbing.
Change the bad-row count in spark_local.py from 12 to 50 and run it again. The rows clean fall to 150, and the sessions with errors change. The pipeline did not break. It told you the data changed. That is what a pipeline is for.
Stop nothing. The Spark jobs end by themselves, the MLflow run lives in the local mlflow.db store, and the lineage manifest lives in lineage.json. Delete the mlflow.db file and the sample files when you are done, or keep them as the start of your own pipeline.
Three questions. No score is saved anywhere. This is just you against the material.
Running a data pipeline is not one skill. It is four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how many tools you can name. They are about whether the pipeline tells you the truth when it breaks. Read them in order. Each level fixes the failure that hurt the most at the level below it.
At the basic level, you follow one batch of events from the object store to the flag, and you name each hop out loud. You run the local Spark job and the local MLflow run, and you read the numbers they print. The failure that ends this level is the first hot key. A clean day feels like the whole job. It is not. It is the easy day.
Do this: run step 0 of the lab. Read the rows in, the rows clean, and the bad rows dropped. Then run step 2 and write down the run id, the data version, and the accuracy. Three numbers, two tools, one day.
At the intermediate level, you set the partition count and watch the largest partition move. You read the Flink watermark and say what a stuck watermark means. You compare two MLflow runs and say which data version trained each one. A run with no data version is a run you cannot repeat, and you stop accepting those.
Do this: run step 1 of the lab, the salted join, and write down the largest partition before and after salting. Then change the salt range from 8 to 2 and run it again. Watch the largest partition grow back. The salt is the fix, and the range is the dose.
At the advanced level, you salt a hot key and prove the largest partition shrank. You name what exactly once costs in Flink and where duplicates still sneak in. You move the features into a feature store so training and serving read the same numbers. The failure that ends this level is the training-serving skew: the model trains on one feature and serves on another, and nobody notices for a week.
Do this: take one feature from the lab model and write down where training reads it and where serving reads it. If the two answers are different places, you have a skew waiting to happen. Move them to one place.
At the master level, the pipeline is a contract. You write data contracts with the team that sends the events: schema, volume, and the dead-letter rule, all in writing. You set alerts on the largest partition, the dead-letter queue, and the watermark, not just on job time. You price the pipeline per gigabyte and you can say what 10x costs before the traffic arrives. The pipeline is a product now, and the on-call rotation treats it like one.
Do this: write one alert for one signal. Alert when one partition passes 10 percent of the rows. Alert when the dead-letter queue grows by more than 1,000 rows in an hour. Alert when the watermark stops moving for five minutes. Three alerts, three causes, zero pages that say only that the job is late.
Spark rebuilds history. Flink answers now. A team that picks one engine for both jobs gets a batch job that is always late or a stream job that cannot look back. The split in this lesson is the honest one: Spark owns the day, Flink owns the last five minutes, and the warehouse is where the two answers meet. When the two answers disagree, the warehouse wins, because the warehouse has the whole day and the stream has five minutes of it.
The fix is salting: add a random number to the hot key, split it across partitions, then roll it back up. The code is in the lab, and the lab prints the largest partition before and after so you can see the fix work. The number to watch is the largest partition, not the job time. Job time hides the skew. The largest partition names it. Adding executors to a skewed job is like adding lanes to a road with one car on it. The car is still the bottleneck.
Flink can promise exactly once inside its own state, with checkpoints and a sink that commits. The promise ends at the sink. A sink that is not idempotent will write a duplicate on a retry, and the warehouse will count it twice. The check is the row count after the write. The count is the contract. If the count is wrong, the promise was broken somewhere past the checkpoint, and the sink is the first place to look.
Three tools, three jobs. DVC records the hash of the data file, so a retrain can name the exact bytes it learned from. MLflow records the parameters and the score, so a result can be repeated. Kubeflow records the steps, so a pipeline can be rerun. A team that has one of the three has a third of the story. The lab shows all three: a DVC stage file you validate by parsing it, an MLflow run you can open in the local SQLite store, and a Kubeflow pipeline file you syntax-check. The stage file is below.
stages:
clean:
cmd: python3 spark_local.py
deps:
- events_sample.csv
- spark_local.py
outs:
- events_clean.parquet
train:
cmd: python3 mlflow_local.py
deps:
- events_clean.parquet
- mlflow_local.py
metrics:
- metrics.json:
cache: false
Read it as a contract. The clean stage depends on the data file and the cleaning code, and it produces the clean file. The train stage depends on the clean file and the training code, and it produces the metrics. Change the data file and DVC knows the clean stage is stale. Change the code and DVC knows too. That is the whole idea: the pipeline knows what it depends on, so a retrain never runs on the wrong bytes by accident.
SHAP and LIME both answer the same question: what did each feature add to this one decision. SHAP spreads the credit by the rules of a fair game, so the contributions add up to the score. LIME builds a small simple model around the one row and reads its weights. The method matters less than the habit: explain the flag before you act on it, and check the top feature for a leak before you trust it. A leak is a feature that knows the answer because it was set after the outcome. The model will love it. Production will not have it. The quiz asks you to catch one.
Spark did not appear from nowhere. It grew out of MapReduce, the older idea: split the work into a map step that turns each row into a key and a value, then a reduce step that gathers every value for one key and combines them. Hadoop was the pair that ran it: HDFS for the storage, MapReduce for the compute. MapReduce writes every step to disk, so a job with five steps reads and writes the disk five times. Spark keeps the working set in memory and only writes when it must, which is why the same job runs faster on Spark. Nobody starts a new pipeline on MapReduce today. The shape is still the shape of every Spark job you will write: map the rows, shuffle by key, reduce the groups. Learn the shape once, and the history makes sense.
At 10x the volume, the shuffle is the bottleneck, and the partition count becomes a setting you tune every week. The largest partition is the alert, and salting becomes a habit, not a fix. At 100x, the ingest lands in a stream first and the batch job reads from the stream, because files that big stop being files and start being a system. The tools do not change. The contracts do: schema, volume, and the dead-letter rule get written down, because at 100x an unwritten contract breaks every day.
The nightly Spark job ran 40 minutes every night for a month. Then one night it ran 6 hours. The morning model trained on yesterday's features, because the fresh ones were not ready. The dashboard flagged a risky session four hours late. Nobody changed the code. One customer ran a sale, and one session id held 40 percent of the day.
Two things. First, the fix. The hot key gets salted before the join, and the largest partition is graphed on the dashboard. The alert fires when one partition passes 10 percent of the rows, not when the job is late. Second, the rule. The job time is no longer the health signal. The largest partition is. A job that runs long is a symptom. A partition that holds 40 percent is the cause. Adding executors to a skewed job is now the example the team uses for a fix that cannot work.
Design the pipeline for 500 GB a day of app events. The events land as JSON lines in an object store. A model flags risky sessions, and every flag needs its reasons on the dashboard within 15 minutes of the session ending. The team is three engineers. The budget is one small cluster and one object store. Name the ingestion, the batch job, the stream job, the feature store, the training pipeline, and the explanation step. Say what breaks first at 10x.
Start with the contract. The team that sends the events writes the schema, and the schema lives in version control. Ingest checks every row and quarantines the bad ones to a dead-letter queue. Nothing is dropped silently. Spark runs once an hour on the landed files, 32 partitions to start, and writes clean features to the warehouse. Flink runs the five-minute window for the live flag, and the flag goes to the dashboard with its SHAP reasons. MLflow logs every run with the DVC data version. Kubeflow retrains nightly and registers only if the score holds.
Then the numbers. At 500 GB a day the Spark hop is the big one: at the simulator formula, 500 GB with 4 partitions is a 5,640 ms Spark hop and a 2,200 ms training hop, so the partition count is 32, not 4, and the largest partition is the alert. The day is 100 million events, so the training job samples 500,000 rows instead of reading the whole day. SHAP explains a random sample of flags, not every flag, because explaining every flag at 1 ms a row is a second job by itself.
Then what breaks first at 10x. The shuffle. Five TB a day means the join needs salted keys from day one, and the Flink window needs a second job just for the hot keys. The ingest lands in a stream first, and the batch job reads from the stream, because files that big stop being files. The dead-letter queue becomes a service of its own, with its own alert, because at 10x the bad rows arrive by the thousand.
Trade-offs to name. Freshness against cost: an hourly Spark job is cheaper than a five-minute one, and the live flag is what covers the gap. Stream complexity against latency: Flink answers now, and Flink is a job that never stops. Explanation coverage against compute: SHAP on every flag is dear, SHAP on a random sample is the compromise, and the sample must be random, not the first hundred.
Saved in this browser, so the Lessons index can show your progress across the course.