Python runs my glue code. It is the first language I reach for when I need a script that works today. But the tools I trust at 2 AM are mostly small compiled binaries, and most of those binaries are written in Go. This lesson builds one of them: a health prober that checks a list of targets at the same time, retries the ones that fail, and writes a JSON report. You will write Go, run Go, and see why concurrency in Go is a line of code and not a library.
Difficulty: Intermediate · Know this first: Lesson 74, Lesson 72, Lesson 162
By the end of this lesson you will have run Go for real, not read about it. First, a working Go install and a hello-world binary that prints its own version. Second, a five-request race that proves concurrency with a stopwatch: about 1,018 ms one at a time, about 202 ms all at once, on this machine. Third, a real CLI prober named probe with flags for targets, workers, timeout, and retries, plus a JSON report a second tool can read. Fourth, three runs of it against a local test server, including the run where one slow target costs 6,003 ms because timeout times retries is the real bill.
What you need: Go 1.23 or newer on your own machine, and a terminal. No cloud account, no API key, no cluster. Every Go program in this lesson was run on this machine with Go 1.23.4, and the outputs shown are the outputs that ran. Your milliseconds will differ. The statuses and the summary will not. Nothing in this lesson runs against a live system from this page. The test server runs on your own machine, on 127.0.0.1.
Go was built for the job DevOps tools do. Start fast. Run as one binary. Do many things at once. Fail loudly. A goroutine costs almost nothing, so starting a thousand of them is a normal thing to do. A channel moves the answer from the worker to the main loop. The compiler catches the typo before the deploy does, because the binary does not exist until the code is right. That is the whole pitch. It is a small pitch, and it is why the tools on my path are mostly Go binaries.
Python does the same job differently. Lesson 74 built the Python prober, and Lesson 162 served an API in Python with FastAPI. Python is the language I reach for when I need a script that works today, and FastAPI is the proof that Python did this first: a real server, a real API, real request handling, in a language that reads like plain English. Go is the second language, not the replacement. You own both, and you pick by the job. The table below is the pick, line by line.
| The job | Python (Lessons 74 and 162) | Go (this lesson) |
|---|---|---|
| Start it | Needs the interpreter, the packages, and the virtual environment. The machine must have Python and the right versions. | One binary. Copy it to the machine and run it. Nothing to install. |
| Do things at once | Threads share one interpreter lock, so CPU work queues. Async works, but you build it by hand with a library. | A goroutine per target, one line each. The runtime schedules them. The code reads like a loop. |
| Fail loudly | An exception jumps to the nearest handler, or to the top. A missed handler is a crash you find later. | Every call returns its error next to its answer. You handle it where it happens, or the compiler will not help you hide it. |
| Ship it | Ship the code and the environment file, and hope the target machine matches. | Ship the binary. The target machine needs nothing. |
| What breaks first | The environment. A package version moves, and the script that worked yesterday fails today. | The timeout you forgot. Go gives you no timeout until you ask for one, and a hung target waits forever. |
Three words carry this lesson. A goroutine is a cheap thread the Go runtime manages. You start one with the word go in front of a function call, and it runs beside the rest of your code. A channel is the inbox the goroutines send answers into. The main loop reads the inbox until it closes, and that is how the results come home. A WaitGroup is the counter that says when the last worker is done, so the inbox can close. That is the whole model. Goroutine, channel, WaitGroup. Everything in the prober is those three words, arranged.
Open a terminal and run go version. If it prints a version, you are ready. If it prints nothing, install Go from go.dev and run it again. Then save the hello-world program from the lab as hello.go and run go run hello.go. The lab shows the output that ran on this machine. Your version line will match. Your CPU count may not. That is fine. The point of this step is small: the toolchain works, and a Go program is a file you run.
The diagram above follows one probe run over five targets. These four paths are the same idea drawn four ways. One run, clean. One run, refused. One run, priced hop by hop. One run, waiting on a slow target. Run each path. One run takes 7.2 seconds across the path, so you can watch each hop light up in turn.
The return rule: the green dot only starts moving after the orange dot reaches the data hop. A return with no request is a lie. The diagram never draws one. The failure replay at the end of Run all four shows the other side of the rule. Run it and watch the orange dot stop at the dead target and never come back.
One probe run over five healthy targets. Flags parse, the file reads, four workers start, the targets answer in 34 ms, and the verdict is written. The retry hop is skipped here because no target needed a second try.
Not run yet.
Same run, one target down. The target refuses the connection, and it refuses fast. The packet stops at the dead target and never comes back, because a refused connection has no answer to return.
Not run yet.
The same run, priced hop by hop at the simulator defaults: 5 targets, 4 workers, 3 second timeout, 1 retry. The trail adds up to 72 ms. The checks hop is the big line. The retry hop is the second bill.
Not run yet.
The same run with one target hanging for 5 seconds. The timeout is 3 seconds, so the checks hop costs 3,000 ms. The verdict still comes, but it comes late, and the retry hop is skipped here because the story is the wait, not the second try.
Not run yet.
Every hop charges rent in milliseconds. The bar below is a worked example for one run of the prober over five targets, not a measurement from a live system, but the shape is honest. The checks hop is the big one, because five targets checked two batches at a time is the real work. The retry hop is the second bill. Move a simulator slider and watch the checks hop and the retry hop move with it. Switch the scenario and watch the slow target take over the whole bar instead.
The budget bar above shows three fixed stories. This simulator lets you write your own. It answers the trade-off at the heart of this lesson: workers cut the wait, but timeout times retries is the bill when one target hangs. You set how many targets to check, how many workers check them, how long one check may wait, and how many times a failed check is retried. Move a slider and watch the checks hop, the retry hop, and the budget total change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: batches are the wait, and the worst hang is the timeout times the tries.
How many URLs are in the targets file. More targets means more work, and the workers share it.
How many goroutines check at the same time. The targets run in batches, and the batches are the wait.
How long one check may wait before it gives up. A hung target waits out the whole timeout, so this number is the worst case per try.
How many times a failed check is tried again. Each retry costs the same as the first try, and a retried timeout costs the timeout again.
Try this: (1) At the defaults, 5 targets, 4 workers, 3 second timeout, 1 retry, the run is 2 batches. The checks hop is 34 ms and the retry hop is 17 ms. The budget total is 72 ms, and the worst hang for one slow target is 6 seconds. (2) Drop to 1 target, 1 worker, 1 second, 0 retries. The checks hop falls to 17 ms and the retry hop to 0 ms. The budget total is 38 ms, and the worst hang is 1 second. (3) Push to 50 targets, 2 workers, 10 second timeout, 3 retries. The run is 25 batches. The checks hop grows to 425 ms and the retry hop to 51 ms. The budget total is 497 ms, and the worst hang for one slow target is 40 seconds. From the defaults, the big run waits 425 ms longer per run and risks 34 seconds more on one hung target. That is the trade, priced.
Formulas, in plain terms. Batches = targets divided by workers, rounded up. Checks hop = batches times 17 ms. Retry hop = retries times 17 ms. Budget total = 21 ms of setup and counting plus the checks hop plus the retry hop. Worst hang = 1 plus retries, times the timeout in seconds. Seventeen milliseconds is the teaching cost of one healthy check. Your targets will differ. The shape will not.
Do this now, on your own machine. You will install Go, prove concurrency with a stopwatch, start a local test server, and run the prober three times. Every program below was run on this machine with Go 1.23.4, and the outputs shown are the outputs that ran. Your milliseconds will differ. The statuses and the summary will not. The order of the OK and FAIL lines also changes run to run, because the checks run at the same time. That is not a bug. That is concurrency, and you can see it in the output.
Install Go from go.dev, then check the version. Save this file as hello.go. It prints a greeting, the Go version, and how many CPUs the machine has.
package main
import (
"fmt"
"runtime"
)
func main() {
fmt.Println("Hello from Go. I am a compiled binary.")
fmt.Println("Go version:", runtime.Version())
fmt.Println("CPUs on this machine:", runtime.NumCPU())
}
go version go run hello.go
Expected output, exactly as it ran on this machine:
go version go1.23.4 linux/amd64 Hello from Go. I am a compiled binary. Go version: go1.23.4 CPUs on this machine: 4
Your CPU count may differ. Your version line should match if you installed the same Go. If go version prints nothing, Go is not on your path yet. Fix the install before you go on. Nothing after this step works without it.
Save this file as race.go. It starts a fake server that sleeps 200 ms on every request, then fetches the same URL five times, once one at a time and once all at once. No flags, no setup. The stopwatch is the lesson.
package main
import (
"fmt"
"net/http"
"net/http/httptest"
"sync"
"time"
)
func fetch(client *http.Client, url string) {
resp, err := client.Get(url)
if err != nil {
fmt.Println("error:", err)
return
}
resp.Body.Close()
}
func main() {
// A fake slow server: every request takes 200 ms on purpose.
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
time.Sleep(200 * time.Millisecond)
fmt.Fprintln(w, "ok")
}))
defer server.Close()
urls := []string{server.URL, server.URL, server.URL, server.URL, server.URL}
client := &http.Client{Timeout: 5 * time.Second}
start := time.Now()
for _, url := range urls {
fetch(client, url)
}
fmt.Printf("one at a time: 5 checks took %d ms\n", time.Since(start).Milliseconds())
start = time.Now()
var wg sync.WaitGroup
for _, url := range urls {
wg.Add(1)
go func(u string) {
defer wg.Done()
fetch(client, u)
}(url)
}
wg.Wait()
fmt.Printf("all at once: 5 checks took %d ms\n", time.Since(start).Milliseconds())
}
go run race.go
Expected output, exactly as it ran on this machine:
one at a time: 5 checks took 1018 ms all at once: 5 checks took 202 ms
Five checks at 200 ms each is 1,000 ms one at a time, and 200 ms all at once. Your numbers will differ by a few milliseconds. The ratio will not. This is the whole reason the prober uses goroutines. The work is the same. The wait is not.
Save this file as server.go. It listens on 127.0.0.1:18080 and serves three kinds of answer: a fast 200, a 404 on /missing, and a 5 second sleep on /slow. It is the target the prober checks. It runs on your machine only.
package main
import (
"fmt"
"net/http"
"time"
)
func main() {
mux := http.NewServeMux()
mux.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "healthy")
})
mux.HandleFunc("/slow", func(w http.ResponseWriter, r *http.Request) {
time.Sleep(5 * time.Second)
fmt.Fprintln(w, "finally")
})
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/missing" {
http.NotFound(w, r)
return
}
fmt.Fprintln(w, "ok")
})
fmt.Println("listening on 127.0.0.1:18080")
if err := http.ListenAndServe("127.0.0.1:18080", mux); err != nil {
fmt.Println("server stopped:", err)
}
}
Save these two targets files. The clean file has five healthy targets. The full file has two healthy targets, one 404, one slow target, and one dead port.
http://127.0.0.1:18080/ http://127.0.0.1:18080/health http://127.0.0.1:18080/a http://127.0.0.1:18080/b http://127.0.0.1:18080/c
Save the clean file as targets-clean.txt and this one as targets.txt.
http://127.0.0.1:18080/ http://127.0.0.1:18080/health http://127.0.0.1:18080/missing http://127.0.0.1:18080/slow http://127.0.0.1:18099/
Start the server in the background, then check it answers:
go run server.go > server.log 2>&1 &
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:18080/health
The curl command prints 200. If it prints nothing, the server is not up yet. Wait a second and try again. Keep this terminal open. The server must keep running for step 3.
Save this file as probe.go. It reads the flags, reads the targets file, starts the worker goroutines, checks each target with a timeout, retries the failures, prints the verdict, and writes the JSON report. Read it once end to end before you run it. The three Go words from the story are all here: goroutine, channel, WaitGroup.
package main
import (
"bufio"
"encoding/json"
"flag"
"fmt"
"net/http"
"os"
"strings"
"sync"
"time"
)
// Result is one target's verdict. No durations are stored here, so the
// JSON report is the same on every run with the same inputs.
type Result struct {
Target string `json:"target"`
Status string `json:"status"`
Attempts int `json:"attempts"`
Error string `json:"error,omitempty"`
}
func check(client *http.Client, url string, retries int) Result {
res := Result{Target: url}
for attempt := 1; attempt <= 1+retries; attempt++ {
res.Attempts = attempt
resp, err := client.Get(url)
if err != nil {
res.Error = err.Error()
continue
}
resp.Body.Close()
if resp.StatusCode != http.StatusOK {
res.Error = fmt.Sprintf("status %d", resp.StatusCode)
continue
}
res.Status = "OK"
res.Error = ""
return res
}
res.Status = "FAIL"
return res
}
func main() {
targetsFile := flag.String("targets", "targets.txt", "file with one URL per line")
workers := flag.Int("workers", 4, "how many targets to check at once")
timeout := flag.Int("timeout", 3, "seconds before one check gives up")
retries := flag.Int("retries", 1, "how many times to retry a failed check")
out := flag.String("out", "report.json", "where to write the JSON report")
flag.Parse()
file, err := os.Open(*targetsFile)
if err != nil {
fmt.Println("cannot open targets file:", err)
os.Exit(1)
}
defer file.Close()
var targets []string
scanner := bufio.NewScanner(file)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if line != "" && !strings.HasPrefix(line, "#") {
targets = append(targets, line)
}
}
if err := scanner.Err(); err != nil {
fmt.Println("cannot read targets file:", err)
os.Exit(1)
}
if len(targets) == 0 {
fmt.Println("no targets in", *targetsFile)
os.Exit(1)
}
client := &http.Client{Timeout: time.Duration(*timeout) * time.Second}
jobs := make(chan string)
results := make(chan Result)
var wg sync.WaitGroup
for i := 0; i < *workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for url := range jobs {
results <- check(client, url, *retries)
}
}()
}
start := time.Now()
go func() {
for _, url := range targets {
jobs <- url
}
close(jobs)
}()
go func() {
wg.Wait()
close(results)
}()
var report []Result
ok := 0
for res := range results {
report = append(report, res)
if res.Status == "OK" {
ok++
fmt.Printf("OK %s (attempt %d)\n", res.Target, res.Attempts)
} else {
fmt.Printf("FAIL %s (attempt %d): %s\n", res.Target, res.Attempts, res.Error)
}
}
fmt.Printf("%d checked, %d ok, %d failed, took %d ms\n",
len(targets), ok, len(targets)-ok, time.Since(start).Milliseconds())
data, err := json.MarshalIndent(report, "", " ")
if err != nil {
fmt.Println("cannot build report:", err)
os.Exit(1)
}
if err := os.WriteFile(*out, data, 0644); err != nil {
fmt.Println("cannot write report:", err)
os.Exit(1)
}
fmt.Println("report written to", *out)
}
Run it against the clean file first:
go run probe.go -targets targets-clean.txt -workers 4 -timeout 3 -retries 1 -out report-clean.json
Expected output, exactly as it ran on this machine:
OK http://127.0.0.1:18080/a (attempt 1) OK http://127.0.0.1:18080/ (attempt 1) OK http://127.0.0.1:18080/health (attempt 1) OK http://127.0.0.1:18080/c (attempt 1) OK http://127.0.0.1:18080/b (attempt 1) 5 checked, 5 ok, 0 failed, took 2 ms report written to report-clean.json
Now run it against the full file, with the same flags:
go run probe.go -targets targets.txt -workers 4 -timeout 3 -retries 1 -out report.json
Expected output, exactly as it ran on this machine:
OK http://127.0.0.1:18080/health (attempt 1) OK http://127.0.0.1:18080/ (attempt 1) FAIL http://127.0.0.1:18099/ (attempt 2): Get "http://127.0.0.1:18099/": dial tcp 127.0.0.1:18099: connect: connection refused FAIL http://127.0.0.1:18080/missing (attempt 2): status 404 FAIL http://127.0.0.1:18080/slow (attempt 2): Get "http://127.0.0.1:18080/slow": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 5 checked, 2 ok, 3 failed, took 6003 ms report written to report.json
Read that output slowly. The refused target fails fast. The 404 fails fast. The slow target costs 3,000 ms, then the retry costs 3,000 ms more. The run takes 6,003 ms because of one target. That is the slow scenario from the diagram, and you just ran it.
Run it once more, with no retries and a 1 second timeout:
go run probe.go -targets targets.txt -workers 4 -timeout 1 -retries 0 -out report-fast.json
Expected output, exactly as it ran on this machine:
OK http://127.0.0.1:18080/ (attempt 1) FAIL http://127.0.0.1:18080/missing (attempt 1): status 404 OK http://127.0.0.1:18080/health (attempt 1) FAIL http://127.0.0.1:18099/ (attempt 1): Get "http://127.0.0.1:18099/": dial tcp 127.0.0.1:18099: connect: connection refused FAIL http://127.0.0.1:18080/slow (attempt 1): Get "http://127.0.0.1:18080/slow": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 5 checked, 2 ok, 3 failed, took 1000 ms
Same failures, one sixth of the wait. The timeout and the retry are the bill, and two flags changed it from 6 seconds to 1 second. Now open report.json. It has no durations in it on purpose, so it is the same on every run with the same inputs:
[
{
"target": "http://127.0.0.1:18080/health",
"status": "OK",
"attempts": 1
},
{
"target": "http://127.0.0.1:18080/",
"status": "OK",
"attempts": 1
},
{
"target": "http://127.0.0.1:18099/",
"status": "FAIL",
"attempts": 2,
"error": "Get \"http://127.0.0.1:18099/\": dial tcp 127.0.0.1:18099: connect: connection refused"
},
{
"target": "http://127.0.0.1:18080/missing",
"status": "FAIL",
"attempts": 2,
"error": "status 404"
},
{
"target": "http://127.0.0.1:18080/slow",
"status": "FAIL",
"attempts": 2,
"error": "Get \"http://127.0.0.1:18080/slow\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
}
]
The order of the entries changes run to run, because the checks finish in a different order each time. The content does not. Stop the test server when you are done. Find its process and kill it, or close the terminal it runs in.
Three questions. No score is saved anywhere. This is just you against the material.
Writing Go for DevOps is not one skill. It is four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how many language features you can name. They are about whether the tool still works when a target hangs at 2 AM. Read them in order. Each level fixes the failure that hurt the most at the level below it.
At the basic level, you run probe by hand against one targets file and you read the verdict. You know what a goroutine is, what a channel is, and why the checks run at the same time. The failure that ends this level is the first slow target. A prober with no timeout waits forever, and you learn that by watching it hang.
Do this: run step 3 of the lab. Run the clean file, then the full file. Write down the two verdict lines. Then change the timeout to 1 and the retries to 0 and run the full file again. Three runs, three verdicts, one page.
At the intermediate level, the prober runs on a schedule and the report feeds something else. You run it every minute from a cron job or a systemd timer, and you alert on any FAIL in the report. The flags are set per environment: a short timeout in the deploy gate, a longer one in the nightly check. The report is JSON, and the alert reads it, not the terminal.
Do this: run the prober with the full file and open report.json. Write a five line script in any language that reads the file and prints the FAIL targets. If the script prints nothing on a clean run and names the dead target on a bad run, the report is doing its job.
At the advanced level, the timeout is not one number for every target. The fast API gets 1 second. The slow report endpoint gets 10. The retry rule is written down: retry a refused connection once, never retry a timeout. The prober becomes a deploy gate, and the gate fails the deploy when a target is down. A gate that cannot fail the deploy is a decoration.
Do this: add a per-target timeout to the targets file format, one extra column, and teach the prober to read it. Then run the full file with the slow target at 1 second and the rest at 3. The verdict should still name the slow target, and the run should take about 1 second, not 6.
At the master level, the prober is a shipped binary with a version number, built in CI on every change, tested against the same test server from the lab. The targets file is versioned, so a bad edit can be rolled back. The report feeds the status page, and the status page shows the last hour. The team still reads the verdict by hand when something breaks. The automation does the watching. The human does the deciding.
Do this: write one test for the prober. Start the test server from the lab inside the test, run the prober against the clean file, and assert the verdict is 5 checked and 0 failed. Run it in CI. A tool with no test is a tool that breaks quietly.
A goroutine is cheap, and cheap things get started without thinking. A goroutine that waits on a channel nobody ever writes to is a leak. It sits there forever, holding its memory, and a thousand of them is a thousand small leaks. The prober closes its jobs channel when the targets run out, and it closes its results channel when the WaitGroup hits zero. Those two closes are the whole leak defense. Every goroutine you start needs an ending you can name. If you cannot name it, the goroutine will outlive the run.
The prober sets its timeout on the HTTP client, and that is enough for one target. At the next level you want a timeout on the whole run, so one slow target cannot hold the verdict hostage. That is what a context is for. You wrap the run in a context with a deadline, and every check reads it. When the deadline passes, the checks stop, even the ones still waiting. Here is the shape of it, using only the standard library:
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
if err != nil {
return Result{Target: url, Status: "FAIL", Error: err.Error()}
}
resp, err := client.Do(req)
if err != nil {
return Result{Target: url, Status: "FAIL", Error: err.Error()}
}
defer resp.Body.Close()
The client timeout protects one check. The context timeout protects the run. Use both. The client timeout says how long one target may take. The context timeout says how long the whole run may take. The run timeout must be shorter than whatever is waiting on the verdict, or the verdict never arrives.
Two goroutines can share a counter with a mutex, and the mutex will keep it correct. The prober does not do that. It sends results into a channel, and one loop counts them. The rule is short: share memory by communicating, do not communicate by sharing memory. A mutex is the right tool when two goroutines must touch the same value. A channel is the right tool when one goroutine produces answers and another consumes them. The prober is the second shape. Reach for the channel first, and reach for the mutex only when the channel does not fit.
Go has no exceptions. Every call that can fail returns its error next to its answer, and you handle it where it happens. The prober checks the error from every request and records it in the report. That is why the report can say the connection was refused instead of saying something failed. An error you ignore is an error you will meet again at 2 AM, with less information. The compiler does not force you to check the error. The incident will.
At 10x the targets, 50 instead of 5, the worker pool is the first thing to tune. Four workers means 13 batches. Twenty workers means 3. The second thing is file descriptors: every open check holds a connection, and a machine has a limit. The third is DNS. Fifty targets on one slow DNS server is fifty checks waiting on the same lookup, and the fix is a resolver with a cache, not more workers. At 100x, 500 targets, the prober stops being one run and becomes a service with a schedule, a store for the reports, and a status page that reads the store. The goroutine model does not change. The number of things around it does. Python did this first in Lesson 162, with a server that answers requests. Go does it as a binary that asks them. Both shapes are honest. Pick by the job.
Effective Go. The concurrency section is the one to read first: goroutines, channels, and the rule about sharing memory. A Tour of Go, Concurrency. The tour runs in the browser, and the channel exercises are short. The net/http package documentation. The client timeout and the context rules live here. All three pages were checked on November 21, 2026. Lesson 162, FastAPI, is the Python contrast: read it when you want to see the same job done as a server instead of a binary.
A prober with no timeout ran as a deploy gate. The deploy waited on the gate, the gate waited on a target that hung, and the target never answered. The deploy sat there for an hour while the team refreshed the page. Nobody was paged, because the gate had not failed. It had not succeeded either. It was still running, and a check that never finishes is a check that never fails. The deploy went through in the end, with the check skipped by hand, and nobody knew whether the target was healthy.
Two things. First, every check gets a timeout, and the run gets a timeout shorter than the deploy wait. A hung check fails the gate instead of holding it. Second, the gate gets a timeout of its own. If the prober runs longer than the gate allows, the gate fails the deploy. A check that never finishes is a failed check, not a pending one. The timeout rule from this lesson is the fix, and it is one flag and one context.
Design a prober for 2,000 targets across 3 regions, checked every minute, with a status page that shows the last hour. The targets file is owned by three teams, one per region. A whole region can go dark at once. The status page must stay up when a region goes down. The alert must page once per region, not once per target. Draw the shape, name the timeouts and the retries, and say what the prober does when a region disappears.
Shard by region. One worker pool per region, one targets file per region, one binary per region if the regions are far apart. Two thousand targets across three regions is about 670 per region. With 20 workers per region that is 34 batches at 17 ms, about 578 ms per check round. That fits inside a minute with room to spare. The pools do not share a channel across regions. Each region writes its own report, and the page reads all three.
Then the timeouts. Timeout 5 seconds per check, retries 0 for the minute check. The minute check is a measurement, not a rescue. Retries belong to the alerting check, which runs after a failure and retries once before it pages. A slow target costs 5 seconds, not 10, because the minute check never retries a timeout. The run gets a context timeout of 30 seconds, so a region full of hung targets cannot hold the round past half a minute.
Then the page and the alert. The reports go to a store the page reads, not to the page directly. The page shows the last hour from the store, so it stays up when a region goes down. The alert reads the store too. One region dark means one page, with the region name in it, not 670 pages. Two failures in a row before the page goes out, so one bad minute does not wake anyone.
Trade-offs to name. Freshness versus load is the big one. Every minute is 2,000 requests a minute, and the targets feel it. Every 5 minutes is a fifth of the load and a page that is 5 minutes stale. Retries versus alert noise is the second. Retrying a timeout halves the false pages and doubles the wait on a real failure. One binary per region versus one binary for all three is the third. Three binaries cost three deploys and survive a region going dark. One binary costs one deploy and dies with the region it runs in.
Saved in this browser, so the Lessons index can show your progress across the course.