Platform Engineering · Lesson 178

Go for DevOps

Python runs my glue code. It is the first language I reach for when I need a script that works today. But the tools I trust at 2 AM are mostly small compiled binaries, and most of those binaries are written in Go. This lesson builds one of them: a health prober that checks a list of targets at the same time, retries the ones that fail, and writes a JSON report. You will write Go, run Go, and see why concurrency in Go is a line of code and not a library.

Difficulty: Intermediate · Know this first: Lesson 74, Lesson 72, Lesson 162

What you will build

By the end of this lesson you will have run Go for real, not read about it. First, a working Go install and a hello-world binary that prints its own version. Second, a five-request race that proves concurrency with a stopwatch: about 1,018 ms one at a time, about 202 ms all at once, on this machine. Third, a real CLI prober named probe with flags for targets, workers, timeout, and retries, plus a JSON report a second tool can read. Fourth, three runs of it against a local test server, including the run where one slow target costs 6,003 ms because timeout times retries is the real bill.

What you need: Go 1.23 or newer on your own machine, and a terminal. No cloud account, no API key, no cluster. Every Go program in this lesson was run on this machine with Go 1.23.4, and the outputs shown are the outputs that ran. Your milliseconds will differ. The statuses and the summary will not. Nothing in this lesson runs against a live system from this page. The test server runs on your own machine, on 127.0.0.1.

The story: the tool I trust at 2 AM is a binary

Go was built for the job DevOps tools do. Start fast. Run as one binary. Do many things at once. Fail loudly. A goroutine costs almost nothing, so starting a thousand of them is a normal thing to do. A channel moves the answer from the worker to the main loop. The compiler catches the typo before the deploy does, because the binary does not exist until the code is right. That is the whole pitch. It is a small pitch, and it is why the tools on my path are mostly Go binaries.

Python does the same job differently. Lesson 74 built the Python prober, and Lesson 162 served an API in Python with FastAPI. Python is the language I reach for when I need a script that works today, and FastAPI is the proof that Python did this first: a real server, a real API, real request handling, in a language that reads like plain English. Go is the second language, not the replacement. You own both, and you pick by the job. The table below is the pick, line by line.

The jobPython (Lessons 74 and 162)Go (this lesson)
Start itNeeds the interpreter, the packages, and the virtual environment. The machine must have Python and the right versions.One binary. Copy it to the machine and run it. Nothing to install.
Do things at onceThreads share one interpreter lock, so CPU work queues. Async works, but you build it by hand with a library.A goroutine per target, one line each. The runtime schedules them. The code reads like a loop.
Fail loudlyAn exception jumps to the nearest handler, or to the top. A missed handler is a crash you find later.Every call returns its error next to its answer. You handle it where it happens, or the compiler will not help you hide it.
Ship itShip the code and the environment file, and hope the target machine matches.Ship the binary. The target machine needs nothing.
What breaks firstThe environment. A package version moves, and the script that worked yesterday fails today.The timeout you forgot. Go gives you no timeout until you ask for one, and a hung target waits forever.

Three words carry this lesson. A goroutine is a cheap thread the Go runtime manages. You start one with the word go in front of a function call, and it runs beside the rest of your code. A channel is the inbox the goroutines send answers into. The main loop reads the inbox until it closes, and that is how the results come home. A WaitGroup is the counter that says when the last worker is done, so the inbox can close. That is the whole model. Goroutine, channel, WaitGroup. Everything in the prober is those three words, arranged.

Do this now, before the diagram

Open a terminal and run go version. If it prints a version, you are ready. If it prints nothing, install Go from go.dev and run it again. Then save the hello-world program from the lab as hello.go and run go run hello.go. The lab shows the output that ran on this machine. Your version line will match. Your CPU count may not. That is fine. The point of this step is small: the toolchain works, and a Go program is a file you run.

Run it as:

When it breaks, I check

One thing to remember

Four paths, one family

The diagram above follows one probe run over five targets. These four paths are the same idea drawn four ways. One run, clean. One run, refused. One run, priced hop by hop. One run, waiting on a slow target. Run each path. One run takes 7.2 seconds across the path, so you can watch each hop light up in turn.

The return rule: the green dot only starts moving after the orange dot reaches the data hop. A return with no request is a lie. The diagram never draws one. The failure replay at the end of Run all four shows the other side of the rule. Run it and watch the orange dot stop at the dead target and never come back.

Path 1: the clean run

Retry: skipped

One probe run over five healthy targets. Flags parse, the file reads, four workers start, the targets answer in 34 ms, and the verdict is written. The retry hop is skipped here because no target needed a second try.

Client0 ms
Parse flags1 ms
Read targets3 ms
Spawn workers2 ms
Check targets34 ms
Verdict6 ms

Not run yet.

Path 2: the refused target

Check targets: skipped

Same run, one target down. The target refuses the connection, and it refuses fast. The packet stops at the dead target and never comes back, because a refused connection has no answer to return.

Client0 ms
Parse flags1 ms
Read targets3 ms
Spawn workers2 ms
Dead targetrefused

Not run yet.

Path 3: the 72 ms trail

Nothing: skipped

The same run, priced hop by hop at the simulator defaults: 5 targets, 4 workers, 3 second timeout, 1 retry. The trail adds up to 72 ms. The checks hop is the big line. The retry hop is the second bill.

Client0 ms
Parse flags1 ms
Read targets3 ms
Spawn workers2 ms
Check targets34 ms
Collect results4 ms
Retry once17 ms
Write verdict6 ms
Write report5 ms

Not run yet.

Path 4: the slow afternoon

Retry: skipped

The same run with one target hanging for 5 seconds. The timeout is 3 seconds, so the checks hop costs 3,000 ms. The verdict still comes, but it comes late, and the retry hop is skipped here because the story is the wait, not the second try.

Client0 ms
Parse flags1 ms
Read targets3 ms
Spawn workers2 ms
Slow target3000 ms
Verdict15 ms

Not run yet.

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one run of the prober over five targets, not a measurement from a live system, but the shape is honest. The checks hop is the big one, because five targets checked two batches at a time is the real work. The retry hop is the second bill. Move a simulator slider and watch the checks hop and the retry hop move with it. Switch the scenario and watch the slow target take over the whole bar instead.

Targets or workers: move the sliders and watch the run

The budget bar above shows three fixed stories. This simulator lets you write your own. It answers the trade-off at the heart of this lesson: workers cut the wait, but timeout times retries is the bill when one target hangs. You set how many targets to check, how many workers check them, how long one check may wait, and how many times a failed check is retried. Move a slider and watch the checks hop, the retry hop, and the budget total change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: batches are the wait, and the worst hang is the timeout times the tries.

5 targets

How many URLs are in the targets file. More targets means more work, and the workers share it.

4 workers

How many goroutines check at the same time. The targets run in batches, and the batches are the wait.

3 seconds

How long one check may wait before it gives up. A hung target waits out the whole timeout, so this number is the worst case per try.

1 retry

How many times a failed check is tried again. Each retry costs the same as the first try, and a retried timeout costs the timeout again.

Try this: (1) At the defaults, 5 targets, 4 workers, 3 second timeout, 1 retry, the run is 2 batches. The checks hop is 34 ms and the retry hop is 17 ms. The budget total is 72 ms, and the worst hang for one slow target is 6 seconds. (2) Drop to 1 target, 1 worker, 1 second, 0 retries. The checks hop falls to 17 ms and the retry hop to 0 ms. The budget total is 38 ms, and the worst hang is 1 second. (3) Push to 50 targets, 2 workers, 10 second timeout, 3 retries. The run is 25 batches. The checks hop grows to 425 ms and the retry hop to 51 ms. The budget total is 497 ms, and the worst hang for one slow target is 40 seconds. From the defaults, the big run waits 425 ms longer per run and risks 34 seconds more on one hung target. That is the trade, priced.

Formulas, in plain terms. Batches = targets divided by workers, rounded up. Checks hop = batches times 17 ms. Retry hop = retries times 17 ms. Budget total = 21 ms of setup and counting plus the checks hop plus the retry hop. Worst hang = 1 plus retries, times the timeout in seconds. Seventeen milliseconds is the teaching cost of one healthy check. Your targets will differ. The shape will not.

Lab: build the prober yourself

Do this now, on your own machine. You will install Go, prove concurrency with a stopwatch, start a local test server, and run the prober three times. Every program below was run on this machine with Go 1.23.4, and the outputs shown are the outputs that ran. Your milliseconds will differ. The statuses and the summary will not. The order of the OK and FAIL lines also changes run to run, because the checks run at the same time. That is not a bug. That is concurrency, and you can see it in the output.

Step 0: install Go and say hello

Install Go from go.dev, then check the version. Save this file as hello.go. It prints a greeting, the Go version, and how many CPUs the machine has.

package main

import (
	"fmt"
	"runtime"
)

func main() {
	fmt.Println("Hello from Go. I am a compiled binary.")
	fmt.Println("Go version:", runtime.Version())
	fmt.Println("CPUs on this machine:", runtime.NumCPU())
}
go version
go run hello.go

Expected output, exactly as it ran on this machine:

go version go1.23.4 linux/amd64
Hello from Go. I am a compiled binary.
Go version: go1.23.4
CPUs on this machine: 4

Your CPU count may differ. Your version line should match if you installed the same Go. If go version prints nothing, Go is not on your path yet. Fix the install before you go on. Nothing after this step works without it.

Step 1: prove concurrency with a stopwatch

Save this file as race.go. It starts a fake server that sleeps 200 ms on every request, then fetches the same URL five times, once one at a time and once all at once. No flags, no setup. The stopwatch is the lesson.

package main

import (
	"fmt"
	"net/http"
	"net/http/httptest"
	"sync"
	"time"
)

func fetch(client *http.Client, url string) {
	resp, err := client.Get(url)
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	resp.Body.Close()
}

func main() {
	// A fake slow server: every request takes 200 ms on purpose.
	server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		time.Sleep(200 * time.Millisecond)
		fmt.Fprintln(w, "ok")
	}))
	defer server.Close()

	urls := []string{server.URL, server.URL, server.URL, server.URL, server.URL}
	client := &http.Client{Timeout: 5 * time.Second}

	start := time.Now()
	for _, url := range urls {
		fetch(client, url)
	}
	fmt.Printf("one at a time: 5 checks took %d ms\n", time.Since(start).Milliseconds())

	start = time.Now()
	var wg sync.WaitGroup
	for _, url := range urls {
		wg.Add(1)
		go func(u string) {
			defer wg.Done()
			fetch(client, u)
		}(url)
	}
	wg.Wait()
	fmt.Printf("all at once: 5 checks took %d ms\n", time.Since(start).Milliseconds())
}
go run race.go

Expected output, exactly as it ran on this machine:

one at a time: 5 checks took 1018 ms
all at once: 5 checks took 202 ms

Five checks at 200 ms each is 1,000 ms one at a time, and 200 ms all at once. Your numbers will differ by a few milliseconds. The ratio will not. This is the whole reason the prober uses goroutines. The work is the same. The wait is not.

Step 2: start the test server and write the targets files

Save this file as server.go. It listens on 127.0.0.1:18080 and serves three kinds of answer: a fast 200, a 404 on /missing, and a 5 second sleep on /slow. It is the target the prober checks. It runs on your machine only.

package main

import (
	"fmt"
	"net/http"
	"time"
)

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintln(w, "healthy")
	})
	mux.HandleFunc("/slow", func(w http.ResponseWriter, r *http.Request) {
		time.Sleep(5 * time.Second)
		fmt.Fprintln(w, "finally")
	})
	mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		if r.URL.Path == "/missing" {
			http.NotFound(w, r)
			return
		}
		fmt.Fprintln(w, "ok")
	})
	fmt.Println("listening on 127.0.0.1:18080")
	if err := http.ListenAndServe("127.0.0.1:18080", mux); err != nil {
		fmt.Println("server stopped:", err)
	}
}

Save these two targets files. The clean file has five healthy targets. The full file has two healthy targets, one 404, one slow target, and one dead port.

http://127.0.0.1:18080/
http://127.0.0.1:18080/health
http://127.0.0.1:18080/a
http://127.0.0.1:18080/b
http://127.0.0.1:18080/c

Save the clean file as targets-clean.txt and this one as targets.txt.

http://127.0.0.1:18080/
http://127.0.0.1:18080/health
http://127.0.0.1:18080/missing
http://127.0.0.1:18080/slow
http://127.0.0.1:18099/

Start the server in the background, then check it answers:

go run server.go > server.log 2>&1 &
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:18080/health

The curl command prints 200. If it prints nothing, the server is not up yet. Wait a second and try again. Keep this terminal open. The server must keep running for step 3.

Step 3: build the prober and run it three times

Save this file as probe.go. It reads the flags, reads the targets file, starts the worker goroutines, checks each target with a timeout, retries the failures, prints the verdict, and writes the JSON report. Read it once end to end before you run it. The three Go words from the story are all here: goroutine, channel, WaitGroup.

package main

import (
	"bufio"
	"encoding/json"
	"flag"
	"fmt"
	"net/http"
	"os"
	"strings"
	"sync"
	"time"
)

// Result is one target's verdict. No durations are stored here, so the
// JSON report is the same on every run with the same inputs.
type Result struct {
	Target   string `json:"target"`
	Status   string `json:"status"`
	Attempts int    `json:"attempts"`
	Error    string `json:"error,omitempty"`
}

func check(client *http.Client, url string, retries int) Result {
	res := Result{Target: url}
	for attempt := 1; attempt <= 1+retries; attempt++ {
		res.Attempts = attempt
		resp, err := client.Get(url)
		if err != nil {
			res.Error = err.Error()
			continue
		}
		resp.Body.Close()
		if resp.StatusCode != http.StatusOK {
			res.Error = fmt.Sprintf("status %d", resp.StatusCode)
			continue
		}
		res.Status = "OK"
		res.Error = ""
		return res
	}
	res.Status = "FAIL"
	return res
}

func main() {
	targetsFile := flag.String("targets", "targets.txt", "file with one URL per line")
	workers := flag.Int("workers", 4, "how many targets to check at once")
	timeout := flag.Int("timeout", 3, "seconds before one check gives up")
	retries := flag.Int("retries", 1, "how many times to retry a failed check")
	out := flag.String("out", "report.json", "where to write the JSON report")
	flag.Parse()

	file, err := os.Open(*targetsFile)
	if err != nil {
		fmt.Println("cannot open targets file:", err)
		os.Exit(1)
	}
	defer file.Close()

	var targets []string
	scanner := bufio.NewScanner(file)
	for scanner.Scan() {
		line := strings.TrimSpace(scanner.Text())
		if line != "" && !strings.HasPrefix(line, "#") {
			targets = append(targets, line)
		}
	}
	if err := scanner.Err(); err != nil {
		fmt.Println("cannot read targets file:", err)
		os.Exit(1)
	}
	if len(targets) == 0 {
		fmt.Println("no targets in", *targetsFile)
		os.Exit(1)
	}

	client := &http.Client{Timeout: time.Duration(*timeout) * time.Second}
	jobs := make(chan string)
	results := make(chan Result)

	var wg sync.WaitGroup
	for i := 0; i < *workers; i++ {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for url := range jobs {
				results <- check(client, url, *retries)
			}
		}()
	}

	start := time.Now()
	go func() {
		for _, url := range targets {
			jobs <- url
		}
		close(jobs)
	}()
	go func() {
		wg.Wait()
		close(results)
	}()

	var report []Result
	ok := 0
	for res := range results {
		report = append(report, res)
		if res.Status == "OK" {
			ok++
			fmt.Printf("OK   %s (attempt %d)\n", res.Target, res.Attempts)
		} else {
			fmt.Printf("FAIL %s (attempt %d): %s\n", res.Target, res.Attempts, res.Error)
		}
	}
	fmt.Printf("%d checked, %d ok, %d failed, took %d ms\n",
		len(targets), ok, len(targets)-ok, time.Since(start).Milliseconds())

	data, err := json.MarshalIndent(report, "", "  ")
	if err != nil {
		fmt.Println("cannot build report:", err)
		os.Exit(1)
	}
	if err := os.WriteFile(*out, data, 0644); err != nil {
		fmt.Println("cannot write report:", err)
		os.Exit(1)
	}
	fmt.Println("report written to", *out)
}

Run it against the clean file first:

go run probe.go -targets targets-clean.txt -workers 4 -timeout 3 -retries 1 -out report-clean.json

Expected output, exactly as it ran on this machine:

OK   http://127.0.0.1:18080/a (attempt 1)
OK   http://127.0.0.1:18080/ (attempt 1)
OK   http://127.0.0.1:18080/health (attempt 1)
OK   http://127.0.0.1:18080/c (attempt 1)
OK   http://127.0.0.1:18080/b (attempt 1)
5 checked, 5 ok, 0 failed, took 2 ms
report written to report-clean.json

Now run it against the full file, with the same flags:

go run probe.go -targets targets.txt -workers 4 -timeout 3 -retries 1 -out report.json

Expected output, exactly as it ran on this machine:

OK   http://127.0.0.1:18080/health (attempt 1)
OK   http://127.0.0.1:18080/ (attempt 1)
FAIL http://127.0.0.1:18099/ (attempt 2): Get "http://127.0.0.1:18099/": dial tcp 127.0.0.1:18099: connect: connection refused
FAIL http://127.0.0.1:18080/missing (attempt 2): status 404
FAIL http://127.0.0.1:18080/slow (attempt 2): Get "http://127.0.0.1:18080/slow": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
5 checked, 2 ok, 3 failed, took 6003 ms
report written to report.json

Read that output slowly. The refused target fails fast. The 404 fails fast. The slow target costs 3,000 ms, then the retry costs 3,000 ms more. The run takes 6,003 ms because of one target. That is the slow scenario from the diagram, and you just ran it.

Run it once more, with no retries and a 1 second timeout:

go run probe.go -targets targets.txt -workers 4 -timeout 1 -retries 0 -out report-fast.json

Expected output, exactly as it ran on this machine:

OK   http://127.0.0.1:18080/ (attempt 1)
FAIL http://127.0.0.1:18080/missing (attempt 1): status 404
OK   http://127.0.0.1:18080/health (attempt 1)
FAIL http://127.0.0.1:18099/ (attempt 1): Get "http://127.0.0.1:18099/": dial tcp 127.0.0.1:18099: connect: connection refused
FAIL http://127.0.0.1:18080/slow (attempt 1): Get "http://127.0.0.1:18080/slow": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
5 checked, 2 ok, 3 failed, took 1000 ms

Same failures, one sixth of the wait. The timeout and the retry are the bill, and two flags changed it from 6 seconds to 1 second. Now open report.json. It has no durations in it on purpose, so it is the same on every run with the same inputs:

[
  {
    "target": "http://127.0.0.1:18080/health",
    "status": "OK",
    "attempts": 1
  },
  {
    "target": "http://127.0.0.1:18080/",
    "status": "OK",
    "attempts": 1
  },
  {
    "target": "http://127.0.0.1:18099/",
    "status": "FAIL",
    "attempts": 2,
    "error": "Get \"http://127.0.0.1:18099/\": dial tcp 127.0.0.1:18099: connect: connection refused"
  },
  {
    "target": "http://127.0.0.1:18080/missing",
    "status": "FAIL",
    "attempts": 2,
    "error": "status 404"
  },
  {
    "target": "http://127.0.0.1:18080/slow",
    "status": "FAIL",
    "attempts": 2,
    "error": "Get \"http://127.0.0.1:18080/slow\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
  }
]

The order of the entries changes run to run, because the checks finish in a different order each time. The content does not. Stop the test server when you are done. Find its process and kill it, or close the terminal it runs in.

Check yourself

Three questions. No score is saved anywhere. This is just you against the material.

Basic to master

Writing Go for DevOps is not one skill. It is four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how many language features you can name. They are about whether the tool still works when a target hangs at 2 AM. Read them in order. Each level fixes the failure that hurt the most at the level below it.

Basic · Run the prober by hand

At the basic level, you run probe by hand against one targets file and you read the verdict. You know what a goroutine is, what a channel is, and why the checks run at the same time. The failure that ends this level is the first slow target. A prober with no timeout waits forever, and you learn that by watching it hang.

Do this: run step 3 of the lab. Run the clean file, then the full file. Write down the two verdict lines. Then change the timeout to 1 and the retries to 0 and run the full file again. Three runs, three verdicts, one page.

Intermediate · Flags, a report, and a schedule

At the intermediate level, the prober runs on a schedule and the report feeds something else. You run it every minute from a cron job or a systemd timer, and you alert on any FAIL in the report. The flags are set per environment: a short timeout in the deploy gate, a longer one in the nightly check. The report is JSON, and the alert reads it, not the terminal.

Do this: run the prober with the full file and open report.json. Write a five line script in any language that reads the file and prints the FAIL targets. If the script prints nothing on a clean run and names the dead target on a bad run, the report is doing its job.

Advanced · Timeouts per target, and a gate that fails

At the advanced level, the timeout is not one number for every target. The fast API gets 1 second. The slow report endpoint gets 10. The retry rule is written down: retry a refused connection once, never retry a timeout. The prober becomes a deploy gate, and the gate fails the deploy when a target is down. A gate that cannot fail the deploy is a decoration.

Do this: add a per-target timeout to the targets file format, one extra column, and teach the prober to read it. Then run the full file with the slow target at 1 second and the rest at 3. The verdict should still name the slow target, and the run should take about 1 second, not 6.

Master · A shipped binary in CI, with a test server

At the master level, the prober is a shipped binary with a version number, built in CI on every change, tested against the same test server from the lab. The targets file is versioned, so a bad edit can be rolled back. The report feeds the status page, and the status page shows the last hour. The team still reads the verdict by hand when something breaks. The automation does the watching. The human does the deciding.

Do this: write one test for the prober. Start the test server from the lab inside the test, run the prober against the clean file, and assert the verdict is 5 checked and 0 failed. Run it in CI. A tool with no test is a tool that breaks quietly.

Go deeper

The goroutine that never returns

A goroutine is cheap, and cheap things get started without thinking. A goroutine that waits on a channel nobody ever writes to is a leak. It sits there forever, holding its memory, and a thousand of them is a thousand small leaks. The prober closes its jobs channel when the targets run out, and it closes its results channel when the WaitGroup hits zero. Those two closes are the whole leak defense. Every goroutine you start needs an ending you can name. If you cannot name it, the goroutine will outlive the run.

Context cancellation: the timeout with a name

The prober sets its timeout on the HTTP client, and that is enough for one target. At the next level you want a timeout on the whole run, so one slow target cannot hold the verdict hostage. That is what a context is for. You wrap the run in a context with a deadline, and every check reads it. When the deadline passes, the checks stop, even the ones still waiting. Here is the shape of it, using only the standard library:

ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()

req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
if err != nil {
    return Result{Target: url, Status: "FAIL", Error: err.Error()}
}
resp, err := client.Do(req)
if err != nil {
    return Result{Target: url, Status: "FAIL", Error: err.Error()}
}
defer resp.Body.Close()

The client timeout protects one check. The context timeout protects the run. Use both. The client timeout says how long one target may take. The context timeout says how long the whole run may take. The run timeout must be shorter than whatever is waiting on the verdict, or the verdict never arrives.

Channels versus mutexes

Two goroutines can share a counter with a mutex, and the mutex will keep it correct. The prober does not do that. It sends results into a channel, and one loop counts them. The rule is short: share memory by communicating, do not communicate by sharing memory. A mutex is the right tool when two goroutines must touch the same value. A channel is the right tool when one goroutine produces answers and another consumes them. The prober is the second shape. Reach for the channel first, and reach for the mutex only when the channel does not fit.

Errors are values, not exceptions

Go has no exceptions. Every call that can fail returns its error next to its answer, and you handle it where it happens. The prober checks the error from every request and records it in the report. That is why the report can say the connection was refused instead of saying something failed. An error you ignore is an error you will meet again at 2 AM, with less information. The compiler does not force you to check the error. The incident will.

What changes at 10x and 100x

At 10x the targets, 50 instead of 5, the worker pool is the first thing to tune. Four workers means 13 batches. Twenty workers means 3. The second thing is file descriptors: every open check holds a connection, and a machine has a limit. The third is DNS. Fifty targets on one slow DNS server is fifty checks waiting on the same lookup, and the fix is a resolver with a cache, not more workers. At 100x, 500 targets, the prober stops being one run and becomes a service with a schedule, a store for the reports, and a status page that reads the store. The goroutine model does not change. The number of things around it does. Python did this first in Lesson 162, with a server that answers requests. Go does it as a binary that asks them. Both shapes are honest. Pick by the job.

Further reading

Effective Go. The concurrency section is the one to read first: goroutines, channels, and the rule about sharing memory. A Tour of Go, Concurrency. The tour runs in the browser, and the channel exercises are short. The net/http package documentation. The client timeout and the context rules live here. All three pages were checked on November 21, 2026. Lesson 162, FastAPI, is the Python contrast: read it when you want to see the same job done as a server instead of a binary.

One time it broke

What broke

A prober with no timeout ran as a deploy gate. The deploy waited on the gate, the gate waited on a target that hung, and the target never answered. The deploy sat there for an hour while the team refreshed the page. Nobody was paged, because the gate had not failed. It had not succeeded either. It was still running, and a check that never finishes is a check that never fails. The deploy went through in the end, with the check skipped by hand, and nobody knew whether the target was healthy.

Timeline

  • 09:00Deploy starts. The gate runs the prober against five targets.
  • 09:01Four targets answer. The fifth hangs. The prober waits, because no timeout was set.
  • 09:05The deploy page still says the gate is running. Someone refreshes it. It still says running.
  • 09:20The team opens the prober logs. The last line is four OK results. The fifth line never comes.
  • 09:25Someone kills the prober by hand. The gate reports nothing, so the deploy is approved by hand instead.
  • 09:30The deploy ships with the check skipped. The hung target stays hung, and nobody knows why.

What changed after

Two things. First, every check gets a timeout, and the run gets a timeout shorter than the deploy wait. A hung check fails the gate instead of holding it. Second, the gate gets a timeout of its own. If the prober runs longer than the gate allows, the gate fails the deploy. A check that never finishes is a failed check, not a pending one. The timeout rule from this lesson is the fix, and it is one flag and one context.

If you remember nothing else

Design exercise

Design a prober for 2,000 targets across 3 regions, checked every minute, with a status page that shows the last hour. The targets file is owned by three teams, one per region. A whole region can go dark at once. The status page must stay up when a region goes down. The alert must page once per region, not once per target. Draw the shape, name the timeouts and the retries, and say what the prober does when a region disappears.

Clarifying questions to ask

  • How fresh must the page be? One minute old is fine, or does it need the last check?
  • What counts as down? One failed check, or two in a row?
  • Who owns the targets file, and what happens when two teams edit it at once?
  • What happens when a whole region goes dark? Who gets paged, and how many times?

Worked approach

Shard by region. One worker pool per region, one targets file per region, one binary per region if the regions are far apart. Two thousand targets across three regions is about 670 per region. With 20 workers per region that is 34 batches at 17 ms, about 578 ms per check round. That fits inside a minute with room to spare. The pools do not share a channel across regions. Each region writes its own report, and the page reads all three.

Then the timeouts. Timeout 5 seconds per check, retries 0 for the minute check. The minute check is a measurement, not a rescue. Retries belong to the alerting check, which runs after a failure and retries once before it pages. A slow target costs 5 seconds, not 10, because the minute check never retries a timeout. The run gets a context timeout of 30 seconds, so a region full of hung targets cannot hold the round past half a minute.

Then the page and the alert. The reports go to a store the page reads, not to the page directly. The page shows the last hour from the store, so it stays up when a region goes down. The alert reads the store too. One region dark means one page, with the region name in it, not 670 pages. Two failures in a row before the page goes out, so one bad minute does not wake anyone.

Trade-offs to name. Freshness versus load is the big one. Every minute is 2,000 requests a minute, and the targets feel it. Every 5 minutes is a fifth of the load and a page that is 5 minutes stale. Retries versus alert noise is the second. Retrying a timeout halves the false pages and doubles the wait on a real failure. One binary per region versus one binary for all three is the third. Three binaries cost three deploys and survive a region going dark. One binary costs one deploy and dies with the region it runs in.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series: Lesson 179
Maths and Statistics for Data Work

Go gave you the tool. Maths gives you the numbers the tool will measure. Linear algebra, probability, hypothesis testing, regression, and A/B testing, taught the same way: plain sentences, runnable code, and a check you can run yourself.

Go to Lesson 179 →