System Design · Lesson 18

API Gateways

Clients should not have to know how many services I run. They should not have to know which one moved, which one is on version two, or which one needs a token in a different shape. One request comes in, one front door takes it, and that front door does the sorting. That front door is the API gateway. This lesson follows one request from a client, through the gateway, to the services that answer it, and shows what happens when a downstream service goes slow, when the gateway starts aggregating calls, and when the gateway itself becomes the bottleneck.

Difficulty: Intermediate · Know this first: Lesson 01, Lesson 06

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in a healthy trip the fan-out hop is the real work and the gateway's own hops are nearly free. Switch the scenario and watch the bar change. In the slow downstream scenario, one service eats almost the whole budget. In the saturated scenario, the gateway itself eats it before any service does real work.

Push the gateway and watch the queue form

The budget above is one fixed trip. This box is yours. Move the sliders and watch what the gateway does when the traffic grows or a service goes slow. I learned this the slow way: the gateway is not magic. It is a box with a capacity, and when the traffic passes that capacity, the queue is where your users wait.

The gateway waits for the slowest parallel call, so this sets the fan-out hop. A slow call also makes the merge do timeout and partial-answer work.

One gateway instance handles about 1,500 requests per second before it starts queuing. Past that, the wait lands on the gateway-in hop.

Traffic above the limit gets a 429 instead of an answer. The limit does not slow the requests that get through. It rejects the rest.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Go deeper

The lesson above is the happy path with one slow service. Production is the rest of it. Here is what I actually spend my time on once the gateway is running.

Trade-offs I weigh every time

A gateway that does everything is a gateway that becomes the system. Auth, rate limiting, routing, transformation, aggregation, logging. Every job I add to the gateway is a job the services never have to do, and also a job the gateway now has to do for every single request. I keep the gateway thin. It routes, it checks identity, it limits, it translates shapes. It does not run business logic. The day the gateway starts deciding prices or applying discounts, I have built a monolith with extra steps.

Aggregation is the same kind of deal. One call in and three calls out is convenient for the client, and it hides the fan-out from them. But it also means the gateway now owns the failure mode of three services at once. I aggregate only when the client genuinely cannot do the fan-out itself, like a mobile app on a slow connection that would pay three round trips. A web client on a fast connection can often call the services itself and skip the aggregation entirely.

Failure modes that do not look like failures

The gateway is a single point of failure, and it knows it. I never run one. I run at least two, behind a load balancer, in more than one zone. The second failure mode is the config. A route table change, a rate limit change, a transform change. All of it deploys in seconds and all of it can break every endpoint at once. I treat gateway config like code. It gets reviewed, it gets tested in staging, and it rolls out to a slice of traffic before it goes everywhere. The third failure mode is the downstream call with no timeout. A service that hangs forever makes the gateway hang forever, and a gateway that hangs holds connections open until it runs out of them. Every downstream call from the gateway gets a timeout that is shorter than the client's patience, and a fallback that is decided in advance.

What changes at 10x

At ten times the traffic, the gateway stops being a box I think about once a year and becomes a capacity number I track every week. I watch requests per second against the queue, not just against CPU. I count the rate limit across all gateway instances with a shared store, because a per-instance limit stops being a limit the moment I add a second instance. I also start caching more at the edge. Validated tokens stay cached. Route tables stay in memory. Anything the gateway has to fetch per request becomes a bottleneck at this scale, so I fetch it once and reuse it.

What changes at 100x

At a hundred times the traffic, one gateway pair is not enough. I run gateways in more than one region, and I route clients to the nearest one before the request ever reaches a service. The config becomes the riskiest part of the whole system, because one bad route now breaks a hundred times more requests per minute. So config changes go through canary rollout, with automatic rollback if the error rate moves. And I stop aggregating at the gateway for the heaviest pages. The fan-out moves to a dedicated aggregation service behind the gateway, so a slow page cannot take the gateway down with it.

Gateway route config I actually use

This is a generic route table, not tied to any one vendor:

# Gateway route table. Config, not code. Reviewed like code.
routes:
  - path: /orders/*
    service: orders
    strip_prefix: false
    timeout_ms: 500
    rate_limit_per_minute: 600
    auth: required
    # Orders is the money path. Short timeout, strict limit.
  - path: /menu/*
    service: menu
    strip_prefix: false
    timeout_ms: 300
    rate_limit_per_minute: 1200
    auth: optional
    # Menu is read-only and cacheable. Looser limit, no login needed.
  - path: /recommendations/*
    service: recommendations
    strip_prefix: false
    timeout_ms: 150
    rate_limit_per_minute: 1200
    auth: required
    fallback: empty_list
    # Slow recommendations must never block the page.
    # 150 ms, then the gateway answers without them.

The timeout on the recommendations route is the whole lesson in one line. 150 milliseconds, then the page goes out without it. I would rather show a page with no suggestions than make a user wait 400 milliseconds for suggestions they did not ask for.

One time it broke

This one stayed with me because the gateway did exactly what it was told to do. The problem was what it was told.

What broke

A deploy changed how the gateway checked tokens. Before the change, the gateway validated tokens locally and kept a short cache, so it only called the identity service when a token was new or the cache had expired. The deploy turned that cache off. Nobody meant to. A config flag that was supposed to be staging-only went out with the release. From that moment, every single request made the gateway call the identity service before it did anything else. The identity service was built for a trickle of cache misses. It got the full firehose instead. It slowed down. The gateway waited on it for every request. The queue grew. Latency climbed across every endpoint at the same time, and the 429s started because the gateway was full of requests waiting on auth.

Timeline

What changed after

Three things, and I still do all three. First, the token cache flag got a guardrail. The gateway refuses to start with the cache off in production, and the flag is now a separate setting that needs a second person's review to change. Second, I added an alert on identity service call volume from the gateway. A tenfold jump in auth calls is now a page, not a dashboard number I might notice. Third, I added a rule for gateway config deploys: they roll out to one gateway instance first, and I watch the queue depth and auth latency for ten minutes before the change goes to the rest. A config that breaks the gateway quietly is a config I want to catch on one instance, not on all of them.

If you remember nothing else

Try it yourself

This is the kind of question I would ask in an interview, and the kind I ask myself before I change a real gateway. Take a minute with it before you read my approach.

The prompt

You run a food delivery app. The client needs one screen that shows a restaurant menu, the user's past orders, and a few recommendations. Those live in three services: menu, orders, recommendations. Peak traffic is 5,000 requests per second. Old versions of the mobile app are still out there and they send the v1 request shape. Design the gateway for this app. Where does auth happen, what do you rate limit, and what happens when the recommendations service goes slow?

Clarifying questions to ask first

I would ask five things before I sized anything. Is the traffic steady or spiky, and how long does the peak last? How many gateway instances do I need so one dying does not take the whole app down? Which endpoints need a login and which are public, like the menu? Can the client retry a failed request safely, or would a retry place the order twice? And how old are the old app versions, and how long do I have to keep supporting their v1 shape?

Rough sizing, worked

5,000 requests per second at peak, and one gateway instance handles about 1,500 before it queues. I never run at full load. 5,000 divided by 1,500 is about 4 instances at the edge. I run 6, across two zones, so one zone dying still leaves 3 instances carrying the peak. The gateway pair from Lesson 06 sits in front of them, so the gateways themselves are not a single point of failure. The rate limit is per user, counted in a shared store across all 6 instances, not per instance. Menu gets a loose limit because it is public and cacheable. Orders gets a strict limit because it is the money path. Recommendations gets a 150 millisecond timeout with a fallback of an empty list, so a slow recommendations service never blocks the menu and the past orders.

Trade-offs to name

I aggregate at the gateway, because a mobile app on a slow connection should not pay three round trips. But I keep the aggregation thin. The gateway stitches the three answers together. It does not decide what to recommend. That logic stays in the recommendations service. I keep the v1 to v2 transform in the gateway, so old app versions keep working while the services move on, and I set a date to drop v1 support instead of carrying the transform forever. I check auth at the edge for orders and recommendations, and I let menu through without a login, because a public menu should not need one. And I set the timeout on every downstream call before I set anything else, because a call with no timeout is a queue that grows until the gateway falls over.

Next in this series
Lesson 19 — Retries, Timeouts, and Idempotency
Coming next