Platform Engineering · Lesson F08

Application Software: The Programs You Actually Use

You never use the kernel directly. You use applications: the browser, the editor, the music player, the server program answering requests in a dark room somewhere. This lesson is about those programs, what they are made of, and how they stand on the operating system underneath them.

Difficulty: Beginner · Know this first: F07 — The Kernel: The Program That Runs the Machine

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

The bar below is a worked example for this lesson, not a measurement. It shows shares of 100 for an application’s attention on a normal day. The shape is the point: the work itself and the asking take most of it. Switch the scenario and watch the talking share grow on a server.

Extra diagram · from scratch to master

What is running on your machine right now

Start with the operating system. Add one application at a time until the machine you use every day is on the map.

6 things on the map1 floor4 applications
The OS
the floor
Browser
pages and tabs
Editor
text and files
Music player
sound
Files
on the disk
You
the user
Request / flowResponse · the way backFailure / dead end

Extra diagram · from scratch to master

A browser asks, a server answers

Follow one request from your browser to a server program and back, before any of the later lessons add the details.

6 stops on the trip1 ask1 answer
You click
in the browser
Browser
asks
The OS
carries it
Network
the road
Server program
answers
Answer
comes back
Request / flowResponse · the way backFailure / dead end

Try it: change the inputs and watch it bend

Open too many apps and watch the memory go. Pick how many apps are open and how much memory the machine has, and see how much is left. The arithmetic is exact. The lesson is the shape: every app costs memory, and the total adds up fast.

6

How many applications are open. Each one here uses 500 MB, a typical size for a browser with a few tabs or an editor with files open.

8

Total memory in the machine. 1 GB is 1024 MB, so 8 GB is 8192 MB.

Try this: (1) 6 apps, 8 GB of memory. The apps use 3000 MB and 5192 MB is free. (2) 20 apps, 8 GB. The apps want 10000 MB, which is more than 8192 MB, so the machine has to slow down or stop something. (3) 6 apps, 4 GB. The apps use 3000 MB and only 1096 MB is free.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

I take the same applications four times here. First I name them. Then I see how they ask. Then I see how they talk to each other. Last I run them for the world.

Basic — The programs you actually use

Definition: Application software is the programs you use: the browser, the editor, the music player, the server program. They stand on the operating system, they ask it for memory, files, and the network, and they never touch the hardware themselves.

Types:

  • A browser: shows pages, talks to servers.
  • An editor: writes and changes text files.
  • A server program: answers requests, with no screen of its own.

Uses: I use this when someone asks what a program is. I point at the browser. That is an application. The kernel underneath is system software.

Dimensions: One machine, a few apps, one user. What breaks first is usually one app, and I notice it when that app stops and the rest keep going. It costs nothing to learn which programs are applications. It costs real time when I restart the machine for one frozen app. The trade is simple: stop the smallest thing that fixes it.

Intermediate — How applications ask

Definition: At this level I see the asking. An application that wants a file, the network, or more memory asks the operating system through a system call. The kernel does it. An app that is slow is often an app waiting for an answer to something it asked for.

Types:

  • Reading a file: ask, wait, get the data.
  • Using the network: ask, wait, get the answer.
  • Showing a window: ask, and the operating system draws it.

Uses: I use this to find why an app is slow. I ask what it is waiting for. The answer is usually a file, the network, or the disk.

Dimensions: A few apps, asking constantly. What breaks is the thing they are waiting for, and I notice it when many apps slow down together. It costs attention to watch what an app waits for. The trade is that waiting is invisible, and the app that waits looks broken when the thing it waits for is the problem.

Advanced — Applications talking to applications

Definition: At this level I see the conversation. Applications talk to each other through the operating system and the network. The browser talks to the server. The editor talks to its helpers. One app asks, another answers. When one stops, the other waits.

Types:

  • Browser to server: a request, an answer, over the network.
  • App to helper: the editor asks a helper to check the code.
  • App to app on one machine: messages carried by the operating system.

Uses: I use this when two apps work together and one stops. I find the one that stopped, because the other one is only waiting.

Dimensions: Many apps, talking constantly. What breaks is the app that stops answering, and I notice it when the app that talks to it waits forever. It costs a timeout to protect the waiting app. The trade is that a timeout costs a little code, and it buys an app that fails fast instead of waiting forever.

The hung program. A program can stop responding while the machine is fine. I find it, I check it is really stuck, and I stop it. The kill command asks a program to stop. If it will not stop, the force option ends it. I try the polite way first.

$ sleep 60 &
[1] 8297
$ ps -o pid,comm -p 8297
    PID COMMAND
   8297 sleep
$ kill 8297
$ ps -o pid,comm -p 8297
    PID COMMAND

Every flag in the commands above, letter by letter.

  • sleep 60: the 60 is seconds. The & runs it in the background, so the prompt comes back at once. Without the &, the terminal waits 60 seconds and I cannot type.
  • ps -o pid,comm: -o picks the columns to show. Here I ask for pid and comm only. Without -o, ps prints its default columns, which are not the ones I want.
  • ps -p 8297: -p picks one process by its ID. Without it, ps lists the processes in this session, and I have to find mine in the list.
  • kill 8297 sends TERM, the polite stop. The program can clean up first. Use the ID from ps. Kill the wrong ID and the wrong program stops.
  • Two more flags worth knowing: kill -9 sends KILL instead of TERM. The program cannot ignore it and cannot clean up. And ps aux is the classic full list, every process on the machine, with user, CPU, and memory.

Check it: run sleep 60 &, then ps -o pid,comm -p $!. $! is the ID of the last background job. Then run kill $! and run ps again. The sleep is gone.

Try it: run the commands above with your own process id. The second ps shows nothing, because the program is gone. If the program is still there after a polite kill, that is when the force option earns its place.

The program that eats memory. One program can use most of the memory while the rest starve. I list programs by memory use and look at the top of the list. The top line is usually the answer to why the machine is slow.

$ ps -o pid,%mem,comm --sort=-%mem | head -5
    PID %MEM COMMAND
     67  3.9 hatch
    756  0.1 hatch-execd
      1  0.0 systemd
     17  0.0 systemd-journal

Every flag in the commands above, letter by letter.

  • ps -o pid,%mem,comm --sort=-%mem: --sort=-%mem sorts by memory, biggest first. The minus means descending. Drop the minus and the smallest comes first, which is rarely what I want.
  • head -5 keeps the first 5 lines. The header counts as one, so I see the 4 hungriest processes.
  • One more flag worth knowing: --sort=-%cpu sorts by CPU instead of memory. When a machine is slow, I sort by CPU first, then by memory. The two sorts point at different culprits.

Check it: run ps -o pid,%mem,comm --sort=-%mem | head -5, then the same line with --sort=%mem. The list flips upside down. Then try --sort=-%cpu.

Try it: run the command above. The first line is the biggest memory user on this machine. If one program sits above 50 percent, that program is your slowdown. Close it or restart it, then run the command again and watch the number fall.

Master — Running applications for the world

Definition: At this level the applications run on machines I never see, serving thousands of people. I watch them through numbers: how many requests they answer, how slow they are, how often they crash. I restart the ones that stop, and I keep spare copies running so one crash changes nothing.

Types:

  • Server programs: answering requests all day, with no user in front of them.
  • Spare copies: two or three copies of the same app, so one can fail.
  • Watching: numbers that tell me an app is slow before the users do.

Uses: I use this when an application serves real users. I do not wait for users to tell me it broke. I watch the numbers, and I keep a spare copy running.

Dimensions: Thousands of users, many copies of the app, machines I never see. What breaks is the copy nobody watched, and I notice it in the numbers before the users notice it in the slowness. It costs real work to run spares and watch numbers. The trade is that the work costs hours, and it buys an application that stays up when one copy fails.

Config drift. Two machines run the same program with different settings, and one of them breaks. The cause is a config file that changed on one machine and not the other. I compare the files with diff before I guess. Diff shows the exact lines that differ, and nothing else.

$ printf 'port=8080\nmode=prod\n' > app-a.conf
$ printf 'port=8080\nmode=dev\n' > app-b.conf
$ diff app-a.conf app-b.conf
2c2
< mode=prod
---
> mode=dev

Every flag in the commands above, letter by letter.

  • printf takes no flags here. It writes the text into the files, so the two config files exist to compare.
  • diff with no flags shows the changed lines. < marks a line from the first file. > marks a line from the second. The 2c2 at the top means line 2 changed to line 2.
  • One more flag worth knowing: diff -u prints the unified format. It adds --- and +++ headers and marks lines with - and +. Patches and code reviews use this format, not the default one.

Check it: run diff app-a.conf app-b.conf, then diff -u app-a.conf app-b.conf. The first shows 2c2 and angle brackets. The second shows the same change with plus and minus lines.

Try it: create the two files above and run diff. One line differs, and that line is the whole bug. The program on machine B runs in dev mode because one line in one file says so. Diff found it in a second. Guessing would have taken an hour.

The app that works for you and not for them. The same program behaves differently for two people because their environments differ. I compare the versions first, then the settings. A version difference is the most common cause, and it is the cheapest to check.

$ python3 --version
Python 3.12.3

Every flag in the commands above, letter by letter.

  • python3 --version: --version prints the version and stops. It runs no code. I run it first when a script fails, because the version is the first thing that differs between two machines.
  • python3 -V is the same thing, shorter. Capital V. Lowercase v means something else, so I type the capital on purpose.

Check it: run python3 --version and python3 -V. Both print Python 3.12.3.

Try it: run python3 --version on your machine and on the machine where the app fails. If the versions differ, that is your first suspect. Same version, same settings, same result. Different version, different result, and now you know where to look.

Go deeper

An application is a process when it runs, and a file when it does not. The browser on your disk is a file. The browser on your screen is a process. That is the difference between a program and a running program, and it matters every time you ask why something is slow.

Every application you use is built from the same pieces: it reads input, it does work, it writes output. The browser reads clicks, does work, shows pages. The server reads requests, does work, sends answers. The shape from the second lesson never went away.

Applications crash. That is normal. What matters is what happens next. A good application saves its work often, so a crash loses minutes, not hours. A good system keeps a spare copy, so a crash loses nothing for the user.

At 10x the users, one copy of the application is not enough. You run two copies, and a load balancer shares the requests between them. The application does not change. The number of copies does.

At 100x, the applications are the system. Hundreds of small programs, talking to each other, each doing one job. That is the architecture the rest of this course is about.

Try it: stop a runaway program. Start sleep 60 in the background, find it with ps, kill it, and check it is gone. The whole exercise takes ten seconds, and it is the exact sequence you will use the first time a real program hangs on you.

$ sleep 60 &
[1] 8297
$ ps -o pid,comm -p 8297
    PID COMMAND
   8297 sleep
$ kill 8297
$ ps -o pid,comm -p 8297
    PID COMMAND

One time it broke

What broke

A browser tab froze, and the person using it restarted the whole machine. The restart lost unsaved work in three other applications, and the frozen tab was frozen again five minutes later, because the page in it was the problem, not the machine.

Timeline

  • 13:00,one browser tab froze. The page in it had stopped answering.
  • 13:01,the whole machine was restarted. Three applications lost unsaved work.
  • 13:05,the machine came back. The browser was opened again. The same page was opened again.
  • 13:10,the tab froze again. The page was the problem all along.
  • 13:11,the tab was closed, and only the tab. The other applications kept their work.

What changed after

Two things. First, stop the smallest thing that fixes the problem. A frozen tab means close the tab. A frozen app means stop the app. The whole machine is the last resort, not the first. Second, ask whether the problem is the app or the thing the app is showing. A frozen page and a frozen browser are different problems with different fixes.

If you remember nothing else

Design exercise

Design the applications for a small team: a browser, an editor, a chat app, and a server program that holds the team’s files. Say what each one is, what it asks the operating system for, and what happens when each one stops.

Clarifying questions to ask

  • Which of these are applications, and what is the operating system underneath them?
  • What does the server program do when nobody is asking it anything?
  • What happens to the team’s files if the server program stops?
  • What happens if the browser freezes? What is the smallest fix?
  • What would you watch to know the server program is healthy?

Rough sizing

Start with the pieces. Four applications, one operating system, one machine each for the team plus one for the server. The server program answers requests all day, so it needs a machine that stays on. The team’s apps need enough memory for a browser with many tabs, about 500 MB each, so 8 GB per laptop is the floor. The server needs a big disk for the files, 1 TB to start.

Trade-offs to name

One copy of the server program is simple and breaks the whole team when it stops. Two copies cost another machine and keep the team working when one stops. The trade is that a spare copy costs money and buys a team that never waits for a restart.

Next in this series
Lesson F09 — Bits, Bytes, and Files