System Design · Lesson 103

Edge Computing

Lesson 08 gave you the cache. Lesson 102 gave you the function. This lesson puts the function where the cache lives. Edge computing is code that runs at the point of presence, the data center nearest the user, instead of in the region where the database lives. One reader on another continent shows the whole shape. The request hits the nearest point of presence. An edge worker runs there, reads the reader's preferences from an edge store, personalizes the page, and answers, without waiting for the region. I learned the edge the way I learn most models in this job. I shipped a worker that was fast everywhere, watched it serve stale data in one place and fresh data in another, and then read the consistency model until the model confessed. This lesson walks one request through the edge, step by step, from a first worker to the deep end: edge versus region trade-offs, consistency, and observability at the edge.

Difficulty: Advanced · Know this first: Lesson 08, Lesson 102

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one personalized page request, not a measurement from a live system, but the shape is honest: in most slow requests I have chased, one or two hops eat almost the whole budget. Step through the lesson or switch the scenario and watch the bar change. The origin fetch doubles the trip without the worker doing anything different. Same code, same reader. Only the hit changed, and the bill doubled.

Try it: tune the edge and watch the total move

The budget bar above shows three fixed stories. This simulator lets you write your own. It models one personalized page request at the edge, with the hit rate, the distance to the origin, the personalization calls, and the traffic all in the mix. Move a slider and watch the origin hop, the regional hop, and the total change in the budget bar, in the hop timings, and in the status line. The numbers are a teaching example, not live measurements. The shape is the lesson: the worker is steady, the origin is the variable, and the hit rate is the multiplier.

20

Queue wait is 0.2 ms times requests minus one. Default queue wait about 4 ms.

80

Origin fetch average is origin distance times the miss share. Default origin fetch 24 ms.

120

Origin fetch on a miss is the distance. Default origin fetch on a miss 120 ms, average 24 ms.

1

Regional API hop is 35 ms times personalization calls. Default regional API hop 35 ms.

Try this: (1) Leave the defaults: 20 requests, 80 percent hit, 120 ms distance, 1 personalization call. The total is about 123 ms. (2) Push the hit rate to 0 percent, the distance to 200 ms, and the personalization calls to 3 with 100 requests. The total is about 385 ms. Same worker, same code. Only the knobs changed. (3) Set everything back, then set the hit rate to 100 percent and watch the origin fetch fall to 0. That is what a hit buys, and the hit rate is the number that matters.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

Edge computing looks like one big idea. It is not. It is four layers stacked on each other, and each layer only makes sense after the one below it. I learned it in this order, and I teach it in this order: get one worker running at one PoP, then wire it to the edge store, then make it survive a stale read and stay observable, then run it across a hundred PoPs with the right boundaries. Skip a layer and the next one reads like magic. Build them in order and nothing in this lesson is magic.

Basic — One worker, one PoP, one job

Definition: edge computing at the basic level is a worker and a point of presence. A worker is a small function that runs at the PoP: one job, one input, one output, no state between calls. A PoP is a small data center near the user, with a cache and a worker runtime. The worker runs when a request arrives at the PoP, and it stops when it returns. The worker is a function in the sense of Lesson 102, and the PoP is the place it runs.

Types:

  • Workers: Cloudflare Workers, Lambda@Edge, and other edge runtimes, the same idea on different platforms.
  • Points of presence, the small data centers near the user.
  • The request and the response, the input and the output of the worker.
  • The transform, the shaping the worker does to the response.
  • The log, the receipt the platform emits for every run.

Uses: when a team has a small job that depends on where the user is. A redirect by country. A header added by language. A cookie set by theme. The first win is that a job that depends on the user's place stops waiting for the region to answer it.

Dimensions: Scale: holds up to a few thousand requests a day without special care. Failure modes: the worker is small and fast, so a worker that does real work hits the time limit and fails; a worker that calls the region on every request is a region function with a longer name. Cost: per request and per millisecond of run time; a worker that runs a thousand times a day costs almost nothing. Trade-offs: you pay the size and time limits before you get any of the speed, so a job that needs real compute is paying a wait it does not want.

Intermediate — The edge store, the origin, and the hit rate

Definition: the intermediate layer is the worker wired to the edge store and the origin. The edge KV is a small store at the PoP: a key, a value, and a fast read. The origin is the region where the real content lives. The hit rate is the share of requests the edge answers without the origin. The worker reads the KV, fetches from the origin on a miss, and writes the fresh value back to the region. The state lives in the services, not in the worker, because Lesson 102 taught that a function has no memory between calls.

Types:

  • Edge KV stores, the small store at the PoP.
  • Origin fetches, the road to the region on a miss.
  • Cache headers, the rules that say what can be cached and for how long.
  • The hit rate, the number that decides how often the origin is paid.
  • The transform, the per-reader shaping that a cache cannot do.
  • The regional API, the call the worker makes only when it must.

Uses: when the page has more than one piece. A page from the origin, preferences from the KV, a subscription from the region, and a transform that puts them together. Each piece has a home, and the worker is the assembler. A page stops being one file from one place and starts being pieces from the places that own them, and a miss stops being a failure and starts being a fetch.

Dimensions: Scale: holds to millions of requests a day, but the hit rate is the number that matters, because every miss pays the origin. Failure modes: a low hit rate means every request pays the ocean, and the edge buys nothing; a KV that is stale serves the old value, and the old value is the price of the edge being a copy; a transform that is cached serves one reader's page to everyone. Cost: per request and per KV read, so a worker that reads the KV on every request pays for every read. Trade-offs: the edge store buys a fast read and costs a copy that can be stale; the origin fetch buys the truth and costs the ocean; the transform buys per-reader pages and costs a worker that has to run on every request.

Advanced — Consistency, deploys, and watching the edge

Definition: the advanced layer is the consistency model, the deploy, and the observability. The edge KV is eventually consistent: a write in the region takes time to reach every PoP, and a PoP can serve the old value until the new one arrives. A deploy to the edge is a deploy to a hundred PoPs at once, and a bad deploy is bad everywhere at the same time. The logs live on the PoPs, and the observability has to collect them into one place, or the edge is a hundred places nobody watches.

Types:

  • Eventual consistency, the model where a write takes time to spread.
  • Stale-while-revalidate, the pattern where the edge serves the old page while it fetches the new one.
  • Edge deploys, one deploy to every PoP at once.
  • Edge observability: logs, metrics, and traces collected from the PoPs.
  • Regional fallbacks, the path the worker takes when the edge store is down.
  • Rate limits and abuse protection at the edge, the gate that stops a flood before it reaches the region.

Uses: when the edge has real traffic and a real on-call rotation. A deploy that must roll out to a hundred PoPs without breaking all of them at once. A write that must be fresh, so the worker reads the region when it matters. A flood that must be stopped at the edge, before it reaches the region. The edge becomes something you operate, not something you deployed once and hoped about.

Dimensions: Scale: the deploy becomes the bottleneck; a deploy to a hundred PoPs is a hundred deploys, and a bad one is bad everywhere; the limit that matters is the smallest one, and it is usually the time limit on the worker. Failure modes: a bad deploy breaks every PoP at once, because the edge has no canary by default; a stale KV serves the old value to a reader who just paid, and the reader sees the old plan; a log pipeline that drops edge logs leaves a hundred PoPs unwatched. Cost: operational; somebody has to own the deploy, review the consistency, and read the logs. Trade-offs: a fast deploy buys speed and costs a blast radius of every PoP at once; a fresh read buys the truth and costs the ocean; a stale serve buys speed and costs a reader who sees the old value.

Master — Edge versus region, and the platform on top

Definition: the master layer is the judgment about where code belongs, and the platform that runs it at scale. Not every job is an edge job. Fast, per-reader work belongs at the edge. Slow, stateful work belongs in the region. The platform team owns the shared pieces: the worker templates, the KV patterns, the CI pipeline that deploys the workers, and the cost reports. The app teams own their workers, and they build on the platform instead of rebuilding it. The design at this layer is a choice, made on purpose, with the trade-offs written down.

Types:

  • Edge versus region decisions, the written rule for where each job lives.
  • Durable Objects and edge state, the way a little state lives at the edge without a region.
  • IoT edge, compute on devices and gateways at the far edge.
  • Worker templates and shared libraries, the platform's starting point for a new worker.
  • Cost allocation by worker and by team, the bill split so each team sees its own spend.
  • The exit plan: the worker code kept portable, so a worker that outgrows the edge can move to the region without a rewrite.

Uses: when there are many workers and the question is no longer how to write one, but where each job belongs. A redirect by country is an edge job. A database write is a region job. A sensor reading is an IoT edge job. The platform team is the team that makes the other teams fast without making them unsafe, and the exit plan is the part that keeps the platform honest.

Dimensions: Scale: holds at thousands of workers and billions of requests a day, because the cost moves from per-worker heroics to shared patterns, and the config moves from hand-built workers to templates. Failure modes: a worker that outgrows the edge and has no exit plan is a rewrite, not a move; a shared KV with no schema is a store where every writer can break every reader; an IoT edge device that loses its link is a device that must keep working offline, or it is a device that stops when the network does. Cost: a platform team; somebody has to run the templates, answer the tickets, and keep the guardrails current. Trade-offs: the edge buys speed and costs limits, staleness, and a deploy that touches every PoP at once; the region buys power and costs the ocean. The choice is not which is better. The choice is which fits the job, and the job is the thing that decides.

Go deeper

The edge and the region are a trade, and the trade is written down. The edge is fast because it is near, and it is small because it has to run everywhere. The region is slow because it is far, and it is big because it has to run everything. The rule I follow is simple: if the job is fast, per-reader, and stateless, it is an edge job. If the job is slow, shared, or stateful, it is a region job. A redirect is an edge job. A database write is a region job. A transform is an edge job. A payment is a region job. The mistake I see most is an edge worker that does region work, because the worker was easy to write and the region was easy to forget.

The edge is eventually consistent, and eventually is a real number. A write in the region takes seconds to reach every PoP, and a PoP can serve the old value until the new one arrives. The question is not whether the edge is stale. The question is whether the staleness matters. A theme change can wait a few seconds. A payment cannot. The pattern I use is stale-while-revalidate: the edge serves the old page at once, and fetches the new one behind the scenes, so the next reader gets the fresh page and the current reader never waits. The pattern that fails is the one that reads the region on every request, because that is a fast worker with a slow answer, and the edge buys nothing.

Observability at the edge is hard because the edge is a hundred places. The logs live on the PoPs, and the PoPs are not yours. The platform collects them into one place, and the collection is the observability. I log three things on every worker run: the PoP that served it, the hit or miss, and the time. The PoP tells me where the reader was. The hit or miss tells me whether the origin was paid. The time tells me whether the worker was fast. A worker with no PoP in the log is a worker I cannot debug, because I do not know where it ran. And I sample the logs, because a worker that runs a billion times a day emits a billion log lines, and the log bill follows the request count.

The far edge is a device, not a data center. A sensor, a gateway, a camera, a machine on a factory floor. The device has a little compute, a little storage, and a link that goes down. The job at the far edge is to keep working when the link is down: read the sensor, decide locally, and send the result when the link returns. The discipline is the same as the PoP: small code, no state that cannot be rebuilt, and a queue for the results. A device that stops when the network does is a device that was never designed for the edge. The edge is the place where the network is the least reliable, and the code has to assume it will fail.

At 10x the traffic, the hit rate becomes the bottleneck. Every miss pays the origin, and the origin has a limit. I raise the hit rate first, by caching more and by fetching less, before I buy a bigger origin. At 100x, the design changes. The single worker becomes a set of workers, one per concern: one for routing, one for the transform, one for the auth check. The deploy becomes a staged rollout, a few PoPs at a time, because a deploy to a hundred PoPs at once is a hundred ways to break at once. The bill gets a budget line of its own, because at that scale the request count is a real slice of the spend, and a bill nobody budgets for is a bill nobody controls. And the consistency model becomes a budget line too: at 100x, the stale reads are a real slice of the support tickets, and a staleness nobody budgets for is a staleness nobody controls.

export default {
  async fetch(request, env, ctx) {
    const readerId = request.headers.get("x-reader-id") || "anon";
    const stored = await env.PREFS.get("prefs:" + readerId, { type: "json" });
    const prefs = stored || {};
    const lang = prefs.lang || "en";
    const theme = prefs.theme || "light";

    const cache = caches.default;
    let page = await cache.match(request);
    let hit = "hit";
    if (!page) {
      hit = "miss";
      const url = new URL(request.url);
      page = await fetch("https://origin.example.com" + url.pathname);
      ctx.waitUntil(cache.put(request, page.clone()));
    }

    let html = await page.text();
    html = html.replaceAll("{{theme}}", theme).replaceAll("{{lang}}", lang);

    return new Response(html, {
      headers: {
        "content-type": "text/html; charset=utf-8",
        "x-edge-pop": (request.cf && request.cf.colo) || "unknown",
        "x-edge-hit": hit
      }
    });
  }
};

One worker, one story. The worker reads the reader's preferences from KV, fast and possibly a few seconds stale. It checks the edge cache for the page, and fetches from the origin only on a miss, caching the plain page for the next reader. It never caches the transformed page, because the transform is per reader, and a cached transform is one reader's page served to everyone. The transform is small: a theme and a language, nothing that needs the region. The response carries two headers: the PoP that served it and the cache status, so the log tells me where the request ran and whether the origin was paid. Read the worker together and you can see the whole trip: KV read, cache check, origin fetch on miss, per-reader transform. Change one of them without reading the other three and you have changed the trip in a way you did not intend.

One time it broke

This one stayed with me because the cause was one line, and the blast radius was the whole edge. A bad rule went out to every PoP at once, and the edge did exactly what it was told.

What broke

In July 2019, Cloudflare deployed a change to its web application firewall, a WAF rule that was meant to block a small attack. The rule contained a regular expression that was far more expensive to run than intended. When the rule went live, it went live on every point of presence at once, because that is how the edge deploys. The expensive expression ran on every request, the CPU on the edge servers spiked, and Cloudflare's network went down for about 27 minutes. The blast radius went past Cloudflare, because so many sites sit behind it, and the sites went down with it. The cause was not a hardware failure. It was one bad rule, deployed everywhere at once, with no canary and no brake.

Timeline

  • 13:42 UTCa WAF rule change is deployed to Cloudflare's edge; the rule contains a regular expression that is far more expensive than intended.
  • 13:44 UTCCPU on the edge servers spikes as the rule runs on every request; the network starts dropping traffic.
  • 13:48 UTCCloudflare posts that its network is degraded; sites behind Cloudflare fail to load.
  • 14:00 UTCengineers identify the bad rule and begin rolling it back; CPU falls as the rule is removed.
  • 14:09 UTCthe network recovers as the rollback completes; most sites are back within minutes.
  • 14:15 UTCCloudflare reports the incident resolved; the outage lasted about 27 minutes.

What changed after

Three things, from Cloudflare's own postmortem. First, the rule deployment was changed so a new rule rolls out to a small set of PoPs first, and a bad rule is caught in the canary instead of in production. A deploy to every PoP at once is no longer the default. Second, the regular expression engine was given a cost limit, so an expensive expression is stopped before it can spike the CPU. A rule that costs too much is a rule that is refused. Third, the lesson every team took home: the edge is a hundred places, and a deploy to a hundred places is a hundred ways to break at once. I check my own deploy process for the same gap now: if a change can go everywhere at once, it needs a canary and a brake, not a hope and a doc. And I keep the edge lesson next to the AWS, GCP, and Azure lessons from Lessons 99 to 101, because all four outages have the same shape: a human change, a missing guardrail, and a blast radius that nobody drew on the whiteboard.

If you remember nothing else

Design exercise

The prompt

Design the edge setup for a news site with 200,000 readers a day, spread across three continents. Pages are personalized by language and theme. The origin lives in one region. The page must answer in under 150 ms for most readers. The bill must stay under $500 a month at the expected load. Sketch the workers, the KV, the cache rules, and the deploy, and say what you would measure.

Clarifying questions to ask

  • How many readers a day, and how many of them are new, because the hit rate follows the repeat readers, not the total?
  • How fresh must the personalization be, and what happens when a reader changes their theme: do they wait, or see the old theme for a few seconds?
  • Is the $500 budget for the edge only, or for the edge plus the origin, and which part of the bill is allowed to grow first?
  • Who owns the worker deploy: one platform team, or does each team deploy its own, and who reviews a deploy that touches every PoP?
  • What happens when a PoP goes down: does the reader fail, or route to the next PoP, and where does that routing live?

Rough sizing, worked with Little's Law

Start with the budget from this lesson. One request takes about 123 ms end to end at the simulator defaults: fixed edge hops about 31 ms, KV read about 4 ms, origin fetch average about 24 ms, regional call about 35 ms, and store write about 25 ms. At 200,000 readers a day, that is about 2.3 requests a second on average, and a peak of about 50 a second in the morning. Little's Law for the sizing: at 50 requests a second, each spending 0.123 seconds in the system, the number inside the system at once is 50 times 0.123, which is about 6 concurrent requests, so the edge has no capacity problem at all, and the limit that matters is the hit rate. At an 80 percent hit rate, one request in five pays the origin fetch of 120 ms, and the average origin cost is 24 ms. At a 50 percent hit rate, the average origin cost is 60 ms, and the total rises to about 159 ms, which is already over the 150 ms target. The hit rate is the design, not the worker. The bill follows the requests: 200,000 a day is 6 million a month, and at about $0.50 per million that is about $3 for the workers, plus KV reads, which is where the real money is. I alert at 80 percent of the budget, and I check the hit rate daily.

Trade-offs to name

A fresh read buys the truth and costs the ocean. A stale serve buys speed and costs a reader who sees the old value. A canary deploy buys safety and costs a slower rollout, because a few PoPs at a time is slower than all of them at once. A worker that calls the region buys the truth and costs the ocean on every request. And the target is the hit rate, not just the latency. A design that answers in 100 ms at a 50 percent hit rate fails this exercise when the origin falls over, and the hit rate is where the safety is.

Next in this series
Lesson 104 — Helm
Coming next