Platform Engineering · Lesson F09

Bits, Bytes, and Files

Everything a computer remembers is made of two values: off and on, zero and one. Bits group into bytes, bytes into files, files into folders. This lesson counts that ladder, exactly, because every size you will ever read in this course is measured on it.

Difficulty: Beginner · Know this first: F08 — Application Software: The Programs You Actually Use

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

The bar below is a worked example for this lesson, not a measurement. It shows the 1024 bytes of a short note shared across the ladder: the bit, the byte, the kilobyte, and the rest. The parts add up to exactly 1024 bytes, which is exactly 1 KB. Switch the scenario to see how the same ladder describes a photo and a video.

Extra diagram · from scratch to master

The ladder: bit, byte, kilobyte, megabyte

Start with one bit. Climb the ladder one step at a time until a megabyte is in reach.

4 rungs1024 bytes in a KB1048576 bytes in a MB
A bit
0 or 1
A byte
8 bits
A kilobyte
1024 bytes
A megabyte
1024 KB
A gigabyte
1024 MB
Request / flowResponse · the way backFailure / dead end

Extra diagram · from scratch to master

Where a file lives: disk, memory, and back

Follow a file from the disk into memory and back again, and see what changes at each stop.

6 stops on the trip1 disk1 memory
On the disk
saved
You open it
the ask
Into memory
being worked on
You change it
memory copy
You save it
back to disk
On the disk
saved again
Request / flowResponse · the way backFailure / dead end

Try it: change the inputs and watch it bend

Convert bytes yourself. Type a number of bytes, or pick a size, and watch it turn into kilobytes, megabytes, and gigabytes. The arithmetic is exact: every step divides by 1024. The lesson is the ladder: 1024, not 1000.

Pick a size

1 KB is 1024 bytes. 1 MB is 1024 KB, which is 1,048,576 bytes. 1 GB is 1024 MB, which is 1,073,741,824 bytes.

Type any number of bytes and the conversions update. Try 2500000 and watch the megabytes come out as 2.38, not a round number.

Try this: (1) 1,048,576 bytes is exactly 1024 KB and exactly 1 MB. (2) 5,242,880 bytes is exactly 5 MB. (3) 2,500,000 bytes is 2441.41 KB and 2.38 MB. Not every byte count lands on a round number.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

I take the same ladder four times here. First I count bits and bytes. Then I read file sizes. Then I plan how much space things need. Last I plan storage for the world.

Basic — Bits and bytes

Definition: A bit is one switch, off or on. A byte is 8 bits, and it can say 256 things. A kilobyte is 1024 bytes. A megabyte is 1024 kilobytes. A gigabyte is 1024 megabytes. Files are bytes with names. Folders are lists of files.

Types:

  • A bit: two values.
  • A byte: 256 values, one letter.
  • A kilobyte: about a thousand letters.
  • A megabyte: a photo. A gigabyte: a film.

Uses: I use this to read file sizes. When a file says 5 MB, I know it is about five million bytes, and I know how long it will take to send.

Dimensions: One file, a few thousand bytes. What breaks first is my sense of size, and I notice it when a file I thought was small turns out to be huge. It costs a minute to learn the ladder. It costs real time when I guess at sizes. The trade is simple: counting in 1024 is strange at first, and it is the only way the numbers work.

Intermediate — Reading file sizes

Definition: At this level I read sizes and know what they mean. A 2 MB photo is 2,097,152 bytes. A 700 MB film is 734,003,200 bytes. I know that memory is small and fast, disk is big and slow, and a file moves between them when I open it and save it.

Types:

  • Text files: kilobytes, a few thousand bytes.
  • Photos: megabytes, a few million bytes.
  • Videos: hundreds of megabytes, or gigabytes.

Uses: I use this when I run out of space, or when a file takes too long to send. I check the size first, because the size tells me what kind of problem it is.

Dimensions: Many files, many sizes. What breaks is the disk filling up, and I notice it when the biggest files are the ones I never look at. It costs a look at the sizes to find them. The trade is that a few big files cost more than a thousand small ones, and deleting the big ones buys the most space.

Advanced — Planning space

Definition: At this level I plan how much space things need. A thousand photos at 5 MB each is 5,000 MB, which is about 4.88 GB. A hundred videos at 500 MB each is 50,000 MB, which is about 48.83 GB. I do the arithmetic before I buy the disk, not after it fills up.

Types:

  • Photos: count them, multiply by 5 MB, and you know the disk you need.
  • Videos: count them, multiply by their size, and you know the disk you need.
  • Backups: twice the space, because a backup is a second copy of everything.

Uses: I use this when I buy a disk or plan a backup. I count what I have, multiply, and buy the size that fits with room to grow.

Dimensions: Thousands of files, gigabytes of space. What breaks is the plan that guessed, and I notice it when the disk fills up months before I expected. It costs ten minutes of arithmetic to plan. The trade is that arithmetic costs minutes, and a disk that is too small costs a migration.

Decimal and binary on the same disk. The same file has two sizes, depending on who counts. du -h counts in 1024s, the way your computer does. du -B 1000 counts in 1000s, the way a disk maker does. The file did not change. The ruler did.

$ printf 'hello\n' > sumtest.txt
$ du -h sumtest.txt
4.0K    sumtest.txt
$ du -B 1000 sumtest.txt
5       sumtest.txt

Every flag in the commands above, letter by letter.

  • printf takes no flags here. It writes hello into the file, so there is something to measure.
  • du -h: -h means human readable. It prints 4.0K instead of a block count. du measures disk use, not file size. A 6 byte file still takes 4.0K on disk.
  • du -B 1000: -B 1000 counts in blocks of 1000 bytes. The same file reads 5, because 6 bytes need one 1000 byte block, and the size is rounded up. Change the number after -B and the count changes with it.
  • One more flag worth knowing: du -s prints a summary, one total line. Without -s, du prints a line for every file inside a folder. For a folder with hundreds of files, that is hundreds of lines.

Check it: run du -h sumtest.txt, then du -B 1000 sumtest.txt, then du -s sumtest.txt. The first prints 4.0K. The second prints 5. The third prints 4, one line, no breakdown.

Try it: run both commands on a file of your own. The numbers differ, and the difference grows with the file. A 1 TB disk shows as 931 GiB for exactly this reason. Know which ruler the number in front of you was measured with.

A file is bytes, not lines. A file that looks like six characters on screen is six bytes on disk, and it takes eight blocks of 512 bytes to store. The size on disk is always bigger than the size of the text, because the disk stores in fixed-size blocks. I check the real size with stat when the number matters.

$ stat -c '%s bytes, %b blocks of %B' sumtest.txt
6 bytes, 8 blocks of 512

Every flag in the commands above, letter by letter.

  • stat -c: -c means print only what the format string asks for. %s is size in bytes. %b is blocks used. %B is bytes per block. Without -c, stat prints everything, and I have to hunt for the number I want.
  • One more flag worth knowing: plain stat sumtest.txt with no flags prints the full record. Size, blocks, permissions, and the access, modify, and change times. I use it when I want the whole story of a file.

Check it: run stat -c '%s bytes, %b blocks of %B' sumtest.txt, then plain stat sumtest.txt. The first prints one line. The second prints the full record.

Try it: run stat on a small file and on a big file. The small file wastes most of its blocks. A thousand tiny files waste a thousand blocks. That is why a disk full of tiny files fills up faster than the bytes suggest.

Master — Storage for the world

Definition: At this level I plan storage for thousands of people. I count in terabytes and petabytes. A petabyte is 1024 terabytes. I plan for copies, because a disk fails and the copy is the only thing standing between a failure and lost data. I plan for growth, because people always store more next year than this year.

Types:

  • Terabytes: 1024 gigabytes, a thousand films.
  • Petabytes: 1024 terabytes, a library of films.
  • Copies: at least two, in different places, because one copy is not a backup.

Uses: I use this when a service stores data for users. I count what each user stores, multiply by the users, add copies, add growth, and then I buy the storage.

Dimensions: Thousands of users, petabytes of data. What breaks is the copy nobody made, and I notice it when a disk fails and the data is gone. It costs real money to keep copies. The trade is that copies cost disks, and they buy the right to lose one without losing the data.

Silent corruption. A file can change without anyone touching it. A bad disk, a bad copy, or a bad cable can flip a byte, and nothing will tell you. I catch it by hashing the file before and after a copy. If the hashes match, the copy is good. If they do not, the copy is bad, and I copy again.

$ printf 'hello\n' > sumtest.txt
$ sha256sum sumtest.txt
5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03  sumtest.txt
$ cp sumtest.txt copy.txt
$ sha256sum copy.txt
5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03  copy.txt

Every flag in the commands above, letter by letter.

  • sha256sum takes no flags here. It prints the hash, then the filename. The copy has the same hash as the original, which proves the copy is exact.
  • cp with no flags copies a file. It does not copy a directory. Try it on a folder and it refuses.
  • One more flag worth knowing: cp -r copies a directory and everything inside it. The -r means recursive. Without it, cp prints cp: -r not specified; omitting directory, and copies nothing.

Check it: run cp sumtest.txt copy.txt and compare the hashes. Then make a folder and run cp folder copy without -r. It refuses. Run it with -r and it works.

Try it: copy a file and hash both copies. The hashes must match exactly. If they do not, the copy is corrupt, and you found out before the bad copy caused damage. Hashing after every important copy is a habit that pays for itself the first time it catches one.

The name is not the file. Renaming a file changes its name and nothing else. The bytes stay where they are, and the hash stays the same. I use this when a file has the wrong name but the right content. Rename it, hash it, and the hash proves nothing was lost.

$ mv copy.txt renamed.txt
$ sha256sum renamed.txt
5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03  renamed.txt

Every flag in the commands above, letter by letter.

  • mv with no flags renames or moves a file. It does not copy. The old name stops existing the moment the new name appears. The hash stays the same, because the file itself never changed.
  • One more flag worth knowing: mv -i asks before overwriting a file that already exists. Without -i, mv replaces the target silently, and the old contents are gone.

Check it: run mv copy.txt renamed.txt, then check the hash again. Then make two files with the same content and run mv -i one.txt two.txt. It asks first. Answer no, and nothing is lost.

Try it: rename a file and hash it before and after. The hash does not change, because the bytes did not change. A rename is safe. An edit is not. The hash is how you tell the difference.

Go deeper

The 1000 versus 1024 difference is real, and it confuses everyone. Disk makers sell a 1 TB disk as 1,000,000,000,000 bytes. Your computer reads it as 931 GB, because it divides by 1024 three times. Nobody is lying. They are counting in different units. Know which one you are reading.

There are two sets of names, and they are the fix for the confusion. The decimal set counts in 1000s: kB, MB, GB, TB, PB. The binary set counts in 1024s: KiB, MiB, GiB, TiB, PiB. The little i is the difference. KiB is not a typo for kB. It is a different unit, and it is the one your computer actually uses.

Decimal, powers of 1000. Disk makers and networks use these.
1 kB = 1,000 bytes
1 MB = 1,000 kB = 1,000,000 bytes
1 GB = 1,000 MB = 1,000,000,000 bytes
1 TB = 1,000 GB = 1,000,000,000,000 bytes
1 PB = 1,000 TB = 1,000,000,000,000,000 bytes

Binary, powers of 1024. Memory and operating systems use these.
1 KiB = 1,024 bytes
1 MiB = 1,024 KiB = 1,048,576 bytes
1 GiB = 1,024 MiB = 1,073,741,824 bytes
1 TiB = 1,024 GiB = 1,099,511,627,776 bytes
1 PiB = 1,024 TiB = 1,125,899,906,842,624 bytes

This lesson writes KB, MB, and GB the way your operating system does: it means the 1024 sizes, the KiB, MiB, and GiB. The quiz does the same. When you see a precise name with an i, it is the binary size. When you see one without, check who wrote it. A disk maker means 1000. Your computer means 1024.

Here is how to calculate. To turn bytes into the decimal sizes, divide by 1000, then by 1000 again, then again. To turn bytes into the binary sizes, divide by 1024, then by 1024 again, then again. To turn a GB into GiB, multiply by 0.9313. To turn a GiB into GB, multiply by 1.0737. To turn bits into bytes, divide by 8.

Three worked examples. A 1 TB disk holds 1,000,000,000,000 bytes. Divide by 1,073,741,824 and you get 931 GiB. That is where the missing space went. A machine with 8 GiB of memory holds 8,589,934,592 bytes. Divide by 1,000,000,000 and you get 8.59 GB. A 1 Gbps link moves 1,000,000,000 bits a second. Divide by 8 and you get 125,000,000 bytes a second, which is 125 MB, or 119.21 MiB.

Where it bites: your operating system shows GiB and labels it GB. A disk maker sells GB. Memory is sold in GiB. Networks quote megabits. Cloud bills usually quote GB, which is the decimal one. Read the unit, not the letter. The letter is the same. The number is not.

A bit is not a small thing. It is the smallest thing. Everything else is counting. When you hear that a network is 100 megabits, that is 100 million bits a second, which is 12.5 megabytes a second, because 8 bits make a byte. Bits and bytes are different units, and mixing them up makes a network look eight times faster than it is.

Files do not have to be stored in one piece on the disk. The disk puts a file’s bytes wherever there is room, and keeps a list of where they are. That is why a file can be fast to read when it was just written and slow to read a year later, after the disk has been filled and emptied many times.

At 10x the size, the ladder is the same. A thousand photos is a few gigabytes. The arithmetic is the same, and the disk you buy is bigger.

At 100x, the ladder reaches petabytes, and the counting is done by machines, not by hand. The units never change. Bytes, kilobytes, megabytes, gigabytes, terabytes, petabytes. Each step is 1024 times the last.

1 bit = 0 or 1
1 byte = 8 bits = 256 values
1 KB = 1024 bytes
1 MB = 1024 KB = 1,048,576 bytes
1 GB = 1024 MB = 1,073,741,824 bytes
1 TB = 1024 GB

That is the whole ladder, written as lines. Read it top to bottom. Each line is 1024 times the one above it.

Try it: the two rulers. Create a small file and run du -h and du -B 1000 on it. The numbers differ. Then run stat and read the byte count. Three commands, three numbers, one file. The file is six bytes. The disk gives it eight blocks. The two size commands disagree because they count in different units.

$ printf 'hello\n' > sumtest.txt
$ du -h sumtest.txt
4.0K    sumtest.txt
$ du -B 1000 sumtest.txt
5       sumtest.txt
$ stat -c '%s bytes, %b blocks of %B' sumtest.txt
6 bytes, 8 blocks of 512

One time it broke

What broke

A disk was full, and the person using the machine was sure it was full of important files. It was not. It was full of videos nobody had watched in years, each one hundreds of megabytes. The important files were a few megabytes in total. The disk was freed in ten minutes by deleting the videos, after checking nobody needed them.

Timeline

  • 16:00,the disk was full. Nothing would save.
  • 16:02,I sorted the folders by size. The biggest folder was videos.
  • 16:05,the videos were hundreds of megabytes each, and nobody had opened them in years.
  • 16:08,I checked with the owner. The videos were not needed.
  • 16:10,the videos were deleted. Half the disk was free.
  • 16:12,the important files saved again. The machine worked.

What changed after

Two things. First, sort by size before you delete anything. A few big files cost more than a thousand small ones, and deleting the big ones buys the most space. Second, a backup before a big delete. The videos were not needed, but checking first cost two minutes, and deleting without checking would have cost far more if I had been wrong.

If you remember nothing else

Design exercise

Plan the storage for a team that takes 200 photos a month, each 5 MB, and records 10 videos a month, each 400 MB. How much space do they need for one year, and how much for a backup copy as well? Show the arithmetic.

Clarifying questions to ask

  • How many bytes is one photo, in exact numbers?
  • How much space do 200 photos a month use in one year?
  • How much space do 10 videos a month use in one year?
  • How much space does the backup copy need?
  • What size disk do you buy, and how much room is left for growth?

Rough sizing

Start with the arithmetic. One photo at 5 MB is 5,242,880 bytes. 200 photos a month is 1000 MB a month, which is 12,000 MB a year, which is about 11.72 GB. One video at 400 MB is 419,430,400 bytes. 10 videos a month is 4000 MB a month, which is 48,000 MB a year, which is about 46.88 GB. Together that is about 58.59 GB a year. A backup copy doubles it to about 117.19 GB. A 500 GB disk holds four years of this, with room to grow.

Trade-offs to name

A bigger disk costs more and buys years of room. A smaller disk costs less and fills up sooner. A backup copy costs double the space and buys the right to lose a disk without losing the work. The trade is that space costs money, and running out of space costs the team its work.

Next in this series
Lesson F10 — Data vs Information