Platform Engineering · Lesson F10

Data vs Information

A sensor writes down a number every second. That number is data. The line on a dashboard that tells you the room is getting hot is information. This lesson is about the gap between the two, and how I cross it every day on call.

Difficulty: Beginner · Know this first: F09 — Bits, Bytes, and Files

Run it as:

You have 86,400 temperature readings from one day and no idea what any of them mean yet. That is where this lesson starts.

When it breaks, I check

One thing to remember

Where the meaning comes from

Turning raw readings into a decision takes effort, and the effort is not spread evenly. The bar below is a teaching example for this lesson, not a measurement from a live system. It shows where the work goes, in 100 units of effort, from the moment a sensor writes a number to the moment a human decides.

Extra diagram · from scratch to master

From one reading to one decision

Start with one sensor. Add one step at a time until a raw number becomes a decision you can defend.

7 steps in the chain1 decision at the end0 guessing
Sensor
writes a number
Raw rows
86,400 a day
Duplicates out
same row twice
Bad rows out
999s and gaps
Time attached
when it happened
One line
31 and rising
Decision
wake or wait
Request / flowResponse · the way backFailure / dead end

Extra diagram · from scratch to master

The same numbers, three readers

One file, three people, three different pieces of information. What counts as information depends on who is reading and what they have to decide.

3 readers1 file1 wrong alert
Raw file
86,400 rows
Engineer
needs the cause
Manager
needs the trend
Alert rule
fires at 30 degrees
Dashboard
the line on the wall
Cried wolf
alert nobody trusts
Request / flowResponse · the way backFailure / dead end

Try it: change the inputs and watch it bend

The simulator below starts with a day of temperature readings and lets you dirty them up. Move the sliders and watch the usable readings shrink. The numbers are a teaching example, not live measurements. The lesson is the shape: every bad row you remove costs you coverage, and every row you keep costs you trust.

120

How many readings the sensor wrote. One every 12 minutes gives 120 a day. One every 6 minutes gives 240.

25

Share of rows the sensor wrote twice. Duplicates are removed first, before anything else.

10

Share of the remaining rows that cannot be true, like a reading of 999 degrees. These are dropped next.

Try this: (1) Set readings to 120, duplicates to 25, bad to 10. You get 81 usable readings out of 120, which is 68 percent. (2) Set duplicates to 50 and bad to 30. You get 8 usable readings out of 24, which is 33 percent. Same sensor, same day. The cleaning decides how much truth is left.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

I take the same idea four times here. First I tell data from information. Then I learn to clean and add context. Then I design the pipeline that does it at scale. Last I run it for the world.

Basic: What data is

Definition: Data is a raw fact written down: a number, a word, a timestamp. Information is that fact after someone adds context and meaning to it, so a person can act on it.

Types:

  • Sensor readings, one number per second.
  • Log lines, one event per line.
  • Form entries, one answer per field.
  • Clicks and page views, one row per action.

Uses: I use this split every time someone hands me a file and asks what it means. I ask what question the file is supposed to answer, and I check whether it can answer it yet.

Dimensions: At small size everything fits in one file and I can read it myself. What breaks first is my own attention, and I notice it when I start skimming instead of reading. It costs nothing to collect data. It costs real time to turn it into information. The trade is simple: collect cheaply, spend the effort on cleaning and context.

Intermediate: Cleaning and context

Definition: At this level I stop trusting raw files. Every row gets checked: is it a duplicate, is it impossible, is it missing its timestamp? Then I attach context: when, where, in what unit. Only then do I look for patterns.

Types:

  • Deduplication rules that say which copy survives.
  • Validity rules that drop impossible values.
  • Enrichment that joins in the missing context.
  • Aggregates that turn many rows into one line.

Uses: I use this when a dashboard disagrees with reality. Nine times out of ten the data was dirty or the context was missing, and the dashboard was faithfully showing a lie.

Dimensions: As volume grows, the cleaning step grows first, because bad rows arrive faster than good ones. What breaks is the rule nobody wrote down, and I notice it when two dashboards show two different answers from the same file. It costs a written rule for every cleaning step. The trade is that written rules are boring to maintain, and they are the only thing that keeps two people from cleaning the same file two different ways.

Advanced: Designing the pipeline

Definition: At this level the chain from data to information is a system, not a chore. Raw events land in one place. Cleaning runs on a schedule. Context is joined automatically. Aggregates are computed once and shared. Every step logs what it dropped and why.

Types:

  • Ingestion that never loses a raw event.
  • Validation that quarantines bad rows instead of deleting them.
  • Enrichment from a source of truth, not a copy.
  • Aggregates with a named owner and a refresh time.

Uses: I use this when more than one team reads the same data. If every team cleans the same file differently, they will argue about whose number is right. One pipeline, one set of rules, one answer.

Dimensions: At ten times the volume the raw store grows fastest and the cleaning rules need the most care. What breaks is the join to the context table, and I notice it when enriched rows start arriving with blank fields. It costs storage for the raw copy and the cleaned copy, plus the habit of keeping both. The trade is that keeping raw events costs disk, and throwing them away costs the ability to ever re-clean them.

Bad rows in good data. A file with a thousand rows can hide two bad ones, and the two bad ones are the ones that break the report. I check every row before I trust the file. The check below finds the rows where the name is missing or the age is not a number.

$ printf 'name,age\nana,34\nbob,not-a-number\n,41\ncara,29\n' > people.csv
$ python3 -c "
import csv
bad = 0
with open('people.csv') as f:
    for n, row in enumerate(csv.DictReader(f), start=2):
        if not row['name'] or not row['age'].isdigit():
            print('line', n, 'bad row:', row)
            bad += 1
print('bad rows:', bad)"
line 3 bad row: {'name': 'bob', 'age': 'not-a-number'}
line 4 bad row: {'name': '', 'age': '41'}
bad rows: 2

Every flag in the commands above, letter by letter.

  • printf takes no flags here. It writes the test rows into the file, so there is bad data to find.
  • python3 -c: -c means run the string that follows as code. The quotes hold the program. Without -c, python3 expects a filename.
  • One more flag worth knowing: python3 -m runs a module by name. python3 -m json.tool reads JSON and prints it indented.

Check it: run python3 -c "print(1+1)". It prints 2.

Try it: run the check above, then fix the two rows in the file and run it again. It prints bad rows: 0. The file is now safe to use. The check took one command. A report built on two bad rows takes a meeting to explain.

The same number, two meanings. A file with four data rows and one header row has five lines. Count the lines and you get five. Count the records and you get four. I count both and I say which one I counted, because the difference is an off-by-one error waiting to happen.

$ wc -l people.csv
5 people.csv

Every flag in the commands above, letter by letter.

  • wc -l: -l counts lines. The file has 5 lines, so the answer is 5. A line is whatever ends with a newline. A file with no trailing newline still counts its last line.
  • Two more flags worth knowing: wc -w counts words, and wc -c counts bytes. On this file, wc -l prints 5, wc -w prints 5, and wc -c prints 45. Three numbers, three different questions.

Check it: run wc -l people.csv, then wc -w people.csv, then wc -c people.csv. The answers are 5, 5, and 45.

Try it: run wc -l on the file above. It says 5. The file holds 4 people and 1 header. When someone says a file has 5 records, ask if they counted the header. Most of the time, they did.

Master: Running it for the world

Definition: At this level data quality is a contract. Every feed has a named owner, a freshness target, and a quality check that pages someone when it fails. Consumers know exactly what they can trust and for how long. Stale information is labelled stale, not served as fresh.

Types:

  • Freshness checks that prove the data arrived on time.
  • Quality checks that prove the data is sane.
  • Lineage that shows where every number came from.
  • Contracts that say what a feed promises and what it does not.

Uses: I use this when a wrong number costs real money or real trust. It is also how I review a system I did not build: I ask who owns the feed, when it last failed a quality check, and who got paged.

Dimensions: At a hundred times the volume the contract is the system, and the storage bill is the price of keeping raw history. What breaks is the thing nobody owns, usually a feed whose owner left the team, and I notice it from the freshness check, not from a user complaint. It costs real operational work: checks, owners, and on-call time. The trade is that the contract costs effort every week, and it buys the right to trust the numbers when a decision is on the line.

A sample that lies. The first ten lines of a file can look clean while line four hundred is broken. I never judge a file by its head. I count the whole file, then I check the whole file. The check in the Advanced section runs over every row for exactly this reason.

$ wc -l people.csv
5 people.csv
$ sort -u people.csv | wc -l
5

Every flag in the commands above, letter by letter.

  • sort -u: -u means unique. It sorts the lines and drops duplicates. Without -u, sort keeps every copy. The pipe into wc -l then counts how many distinct lines there are.
  • One more flag worth knowing: sort -n sorts by number. Without it, sort compares text, so 10 comes before 2. That is the classic sort surprise, and -n is the fix.

Check it: run printf '2\n10\n1\n2\n' | sort. It prints 1, 10, 2, 2. Then run it with sort -n. It prints 1, 2, 2, 10. Then run it with sort -u. It prints 1, 10, 2.

Try it: run both commands. sort -u removes duplicate lines. If the two counts differ, the file has duplicates, and the duplicates are inflating every total built from it. Same count means no exact duplicates. It does not mean the data is clean. It means one specific problem is absent.

Go deeper

Garbage in, garbage out is not a slogan. It is the law of this whole lesson. A dashboard built on dirty rows will show you a confident lie, and confidence is what makes the lie dangerous. I check the data before I check the dashboard.

Timestamps are the cheapest context there is. A reading with no time attached cannot be ordered, cannot be compared, and cannot be trusted in a timeline. When I design any feed, the timestamp is the first field I demand, before the value itself.

One metric, one owner. Every number that matters on a dashboard needs a person who owns it. If nobody owns it, nobody fixes it when it goes wrong, and it quietly becomes a lie that everyone quotes.

A fact and a claim are different things. The sensor read 31 degrees is a fact. The room is overheating is a claim. Claims need evidence, and the evidence is the context around the fact. I keep the two apart in my head, especially when I am tired.

At 10x the volume, the cleaning step breaks first. More rows mean more bad rows, and the rules that handled a thousand rows a day will choke on ten thousand. I plan for the cleaning to be the bottleneck, and I give it its own budget and its own alerts.

At 100x, the design changes. Raw events go to cheap storage that nobody queries directly. Cleaning and enrichment run as a pipeline. Aggregates are computed once and served to everyone. The raw copy is kept, because the day will come when a rule was wrong and the whole history needs re-cleaning.

2026-11-19T14:00:01Z temp=31.0
2026-11-19T14:00:02Z temp=31.0
2026-11-19T14:00:03Z temp=999.0

{ "time": "2026-11-19T14:00:01Z", "room": "server-room-a", "unit": "celsius", "temp": 31.0, "sensor": "s-04" }

What the snippet means: the top lines are raw data, three readings with no room, no unit, and one impossible value. The bottom line is the same reading after context was added. Same number, 31.0. Only the bottom one can be trusted, because only it says what it is, where it is, and when it happened.

Try it: find the two bad rows. Build the four-person file from the Advanced section and run the check. It finds line 3 and line 4. Fix both rows, run the check again, and it prints bad rows: 0. The check is the same. The data changed. That is the whole job of validation. Same test, clean data.

$ python3 -c "
import csv
bad = 0
with open('people.csv') as f:
    for n, row in enumerate(csv.DictReader(f), start=2):
        if not row['name'] or not row['age'].isdigit():
            print('line', n, 'bad row:', row)
            bad += 1
print('bad rows:', bad)"
line 3 bad row: {'name': 'bob', 'age': 'not-a-number'}
line 4 bad row: {'name': '', 'age': '41'}
bad rows: 2

One time it broke

What broke

The dashboard said the server room was fine for two days. It was not fine. The temperature sensor had stuck at 21.0 degrees, and the dashboard faithfully showed 21.0, hour after hour, green all the way. The room was actually at 34 degrees by the time someone walked in and felt the heat. The data was wrong in the most dangerous way possible: it looked right.

Timeline

  • Mon 09:12,the cooling unit in the server room stops. Nobody notices. The sensor keeps writing 21.0.
  • Mon 18:40,I glance at the dashboard before leaving. Green. 21.0 degrees. I go home.
  • Tue 08:05,the morning check shows 21.0 again. Flat line for 23 hours. I almost scroll past it.
  • Tue 08:20,I stop. A flat line for 23 hours is not a healthy room. It is a dead sensor. I walk to the room.
  • Tue 08:31,the room is hot enough to feel through the door. A handheld thermometer reads 34 degrees.
  • Tue 09:10,cooling restarted, sensor replaced, and the stuck readings are marked bad in the log so nobody trusts them later.

What changed after

Three things. First, a staleness check: if a sensor reports the exact same value for 30 minutes, the dashboard flags it as suspicious, not healthy. Second, a second sensor in the same room, so one flat line can be checked against another. Third, a rule I now follow on every dashboard: a flat line is a warning, not a comfort. Healthy systems wobble.

If you remember nothing else

Design exercise

Design a temperature monitor for a small server room. One sensor writes a reading every 10 seconds. You need a dashboard line and a page to the on-call engineer when the room is genuinely overheating. False pages are expensive: every false page teaches the team to ignore the next one.

Clarifying questions to ask

  • How often must the sensor write for the dashboard line to be useful, and how often is just noise?
  • What counts as overheating: one reading above 30 degrees, or a sustained rise? Say the rule in one sentence.
  • What happens when the sensor writes 999, or writes nothing for an hour?
  • How do you tell a hot room from a broken sensor?
  • Who owns the alert rule, and who fixes it when it pages wrongly?

Rough sizing

One reading every 10 seconds is 6 a minute, 360 an hour, 8,640 a day. A month of readings is about 260,000 rows, which is small for any database. The storage is not the problem. The problem is the rule. I set the page rule to three readings above 30 degrees in a row, which is 30 seconds of evidence, not one stray spike. And I add a staleness check: no reading for 5 minutes pages as a sensor fault, not as a temperature problem.

Trade-offs to name

Paging on one reading buys speed and costs false pages. Waiting for three readings buys accuracy and costs 30 seconds. A low threshold buys early warning and costs sleep. A high threshold buys quiet nights and costs a hot room discovered late. And the dashboard line is not the alert. The line is for humans looking. The alert is for machines deciding. Keep them separate.

Next in this series
Lesson F11 — Talking to the Computer: Your First Commands