System Design · Lesson 125

Designing APIs: REST, gRPC & GraphQL

Lesson 18 gave you the front door: the gateway that routes, authenticates, and rate limits. This lesson gives you what lives behind the door. An API is a promise about shape: what a caller asks for, what it gets back, and what happens when it asks wrong. There are three shapes worth knowing. REST says one resource, one URL, and the client makes one call per thing it needs. gRPC says one contract, typed and fast, and the client and the server agree on it in advance. GraphQL says one query, and the client asks for exactly the fields it wants. I learned this the way I learn most tools in this job. I watched a mobile app make eleven calls to draw one screen, because the API was REST and the screen needed eleven things, and then I watched a team break every old app in the field by renaming one field. This lesson walks one request through all three shapes, step by step, and then teaches you the parts that outlive the paradigm: versioning, pagination, errors, idempotency, and how to change an API without breaking the people who use it.

Difficulty: Advanced · Know this first: Lesson 18 (API Gateways)
Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one client request for a page of orders, not a measurement from a live system, but the shape is honest: in a healthy call the handler eats most of the budget, and in a breaking change the error hop eats the extra, because a failure that is mapped, logged, and retried costs more than a success. Step through the lesson or switch the scenario and watch the bar change. The GraphQL trip is the lesson in one picture. It costs about the same as one REST call and delivers three things, because one round trip to the data beats three round trips to three handlers.

Try it: size the API and watch the bytes move

The budget bar above shows three fixed stories. This simulator lets you write your own. It models an API serving a mobile app, with a page size, a cache, and a client that asks for more fields than it needs. Move a slider and watch the response size, the bandwidth, and the verdict change. The numbers are a teaching example, not live measurements. The shape is the lesson: the bandwidth is the request rate times the response size, the cache is the cheapest way to cut both, and over-fetching is a tax the client pays on every call.

1,000

The bandwidth is the requests that miss the cache, times the response size, every second.

20 items

Items per page. Each item is about 2 KB, so the response is the page size times 2 KB, times the over-fetch factor.

50%

The response grows by that share, and the handler does more work to fill them. A 50 percent over-fetch makes the response 1.5 times as big and the handler 1.5 times as slow.

60%

The cache answers that share of requests, so only the rest reach the handler and the network. A 60 percent hit rate means 40 percent of requests do the work.

Try this: (1) Leave the defaults: 1,000 requests a second, page 20, 50 percent extra fields, 60 percent cache. The bandwidth is about 23 MB a second and the handler takes 60 ms. (2) Push the requests to 10,000 a second, the page to 100, the extra fields to 200 percent, and the cache to 0. The bandwidth jumps to about 5,859 MB a second and the handler takes 120 ms. Same API, same data. Only the knobs changed. (3) Set everything back, then set the cache to 95 percent, the page to 10, and the extra fields to 0. The bandwidth falls to about 0.1 MB a second and the handler takes 40 ms. That is what a tight API looks like, and the cache did most of the work.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

API design looks like one choice: REST, gRPC, or GraphQL. It is not. It is four layers stacked on each other, and each layer only makes sense after the one below it. I learned it in this order, and I teach it in this order: one resource and one call, then the rules for paging and failing, then the choice between the three shapes, then changing the API without breaking anyone. Skip a layer and the next one reads like magic. Build them in order and nothing in this lesson is magic.

Basic — One resource, one call, one answer

An API at the basic level is a way for one program to ask another program for something. In REST the something is a resource, and the resource has a URL: /orders/42 is order 42. The client uses a method: GET to read it, POST to create one, PUT to replace it, DELETE to remove it. The server answers with a status code and a body. A 200 means it worked. A 404 means the resource is not there.

Types:

  • Resources, named by nouns, not verbs: /orders, not /getOrders.
  • Methods: GET, POST, PUT, PATCH, DELETE.
  • Status codes: 200, 201, 400, 404, 500.
  • JSON, the body format, readable and flexible.

Uses: when a team is exposing its first API. One resource, a handful of endpoints, and a client that can read them. The first win is that two programs can talk to each other without sharing a database, and the team that owns the data can change the database without breaking the team that reads it.

Scale: holds for one team and a handful of endpoints, with a handful of callers. Failure modes: a resource named by a verb, which confuses every caller; a GET that changes data, which breaks every cache that assumed GET was safe; and an error with no code, which the client cannot switch on. Cost: one server, one client, and the discipline to name things by nouns. Trade-offs: you get a simple contract and nothing else. No paging, no versioning, no rate limiting. A simple API that grows without those is an API that breaks the first time it grows.

Intermediate — Paging, errors, and doing the same thing twice safely

The rules that make an API usable at real size. Pagination says how a client reads a long list: a page size, and a cursor that says where the next page starts. Errors say how the API fails: a code the client can switch on, a message a human can read, and no stack trace. Idempotency says what happens when the client asks twice: an idempotency key on the request, and the server remembers the key, so the second ask returns the first answer instead of doing the work twice.

Types:

  • Pagination: offset and limit, or a cursor, which is the one that holds up at scale.
  • Error codes: a stable code per failure, plus the HTTP status.
  • Idempotency keys, for POSTs that must not run twice.
  • Timeouts and retries, with a limit, so a slow call does not hang forever.
  • ETags and cache headers, so the client does not download the same answer twice.

Uses: when the API has real callers and real data. A list with a thousand rows needs paging, or the response is a megabyte. A payment needs an idempotency key, or a retry charges twice. An error needs a code, or the client cannot tell a bad request from a dead server. The point of this layer is that the API stops being a demo and starts being a contract.

Scale: holds to hundreds of callers and thousands of requests a second. Failure modes: offset pagination on a list that changes, which skips rows and repeats rows; an idempotency key the server never stores, which is a key that does nothing; and a retry with no limit, which turns one slow call into a pile of slow calls. Cost: the cost is design. Each rule is a decision the team has to make once and keep, and a rule changed later is a breaking change. Trade-offs: cursor paging buys correctness and costs a cursor the client has to keep. Idempotency buys safety and costs storage for the keys. A good error buys debuggability and costs a contract the team has to write down.

Advanced — REST, gRPC, and GraphQL: picking the shape

The choice between the three paradigms, and the choice is about fit, not fashion. REST is one resource per URL, stateless, cacheable, and readable, and the client pays one round trip per thing it needs. gRPC is a typed contract in Protocol Buffers, binary on the wire, fast, and streaming, and the client and the server agree on the shape when the code is generated. GraphQL is one query, and the client asks for exactly the fields it wants, so one round trip can fetch three things, and the server pays to parse and shape the answer.

Types:

  • REST: resources, methods, status codes, JSON, cacheable by HTTP.
  • gRPC: Protocol Buffers, service definitions, unary and streaming calls, HTTP/2.
  • GraphQL: a schema, queries, mutations, resolvers, and a single endpoint.
  • The gateway in front of all three, which Lesson 18 covered: routing, auth, rate limiting, and aggregation.

Uses: REST when the callers are many, unknown, and need to cache: a public API, a mobile app, a web frontend. gRPC when the callers are services I own, the traffic is internal, and the speed matters: service to service calls, streaming, and typed contracts that break the build when they change. GraphQL when the client is a screen that needs many things at once, and the round trips cost more than the parsing: a mobile app on a slow connection, a dashboard that stitches five services together. The point of this layer is that the shape follows the caller, not the fashion of the year.

Scale: holds to thousands of callers, because each paradigm scales differently: REST scales on the cache, gRPC scales on the binary wire, GraphQL scales on the resolver batching. Failure modes: REST that makes a client call eleven times for one screen; gRPC used for a public API, where the callers cannot generate the client; GraphQL with no query cost limit, where one query asks for a thousand nested items and the server does all of it. Cost: the cost is the gateway. Three paradigms mean three kinds of rate limiting, three kinds of error, and three kinds of client, and the gateway has to speak all three. Trade-offs: REST buys simplicity and costs round trips. gRPC buys speed and costs a contract the callers must generate. GraphQL buys one round trip and costs parsing, shaping, and a query language the team has to learn. There is no best shape. There is only the shape that fits this caller.

Master — Changing the API without breaking anyone

Change over time. An API is a promise to strangers, and strangers cannot be updated from the server. Schema evolution is the discipline of changing the promise without breaking it: add fields, never remove them; add endpoints, never rename them; deprecate with a date, and remove only when the old callers are gone. A breaking-change policy is the written rule for what counts as breaking, who approves it, and how long the old version lives. The gateway is where the policy is enforced: old versions route to old handlers, new versions route to new ones, and a transform in the gateway can translate between them for a while.

Types:

  • Additive changes: new fields, new endpoints, new optional parameters.
  • Deprecation: a field marked deprecated, with a date and a replacement.
  • Versioning: in the URL, in a header, or in the contract version.
  • The sunset: the date the old version stops answering, announced in advance.
  • Contract testing: a test that fails when the server breaks the shape the clients depend on.
  • The gateway transform: a v1 to v2 translation that lives in the gateway, not in the services.

Uses: when the API has callers I cannot update. A mobile app in the field. A partner's integration. A service owned by another team. The point of this layer is that the API can grow without breaking, and the team that owns it can say, in writing, what breaking means and how long the old version will live.

Scale: holds at the size of a whole platform, because versions multiply: two versions of the API means two handlers, two tests, and two sets of docs. Failure modes: a field removed without a version bump, which is the breaking scenario in this lesson; a deprecation with no date, which is a promise nobody keeps; and a gateway transform that nobody owns, which rots until the day it translates wrong. Cost: the cost is time. Every version that lives is a version that has to be tested, monitored, and eventually killed, and a version that is never killed is a version the team owns forever. Trade-offs: additive change buys safety and costs a response that grows forever, because the old fields never leave. Versioning buys a clean break and costs running two APIs at once. A sunset buys a clean ending and costs a date the team has to keep, even when a caller begs for one more month. There is no free change. There is only a choice about who pays for it, and when.

Go deeper

Schema evolution has four rules, and I have never needed a fifth. Add fields, never remove them. Add endpoints, never rename them. Make new fields optional, never required. And when a field has to die, deprecate it first, with a date and a replacement, and remove it only after the old callers are gone. The rules are simple, and the reason they are simple is that the callers are strangers. A mobile app in the field cannot be updated from the server, and a partner's integration cannot be fixed by a deploy on my side. Every rule in this list exists because someone broke it once and the break was permanent until every caller updated.

The gateway is where the versioning lives, and the services stay clean. The gateway routes v1 to the v1 handler and v2 to the v2 handler, and the services never see a version they do not understand. When a transform is needed, it lives in the gateway too: a v1 request shape turned into a v2 shape before the service sees it, and a v2 answer turned back into v1 before the client sees it. I keep the transform thin. It renames a field and moves on. It does not decide what to recommend or what to charge. That logic stays in the service. And the transform gets a sunset date, written down, because a transform with no date is a second API the team owns forever.

A breaking-change policy is a document, and it says three things. What counts as breaking: removing a field, renaming a field, changing a type, changing an error code, and making an optional field required. Who approves a break: one person, named, not a committee, because a committee approves everything and owns nothing. And how long the old version lives: a number of months, announced in advance, with a sunset header on every response from the old version that says when it dies. I publish the policy where the callers can read it, because a policy the callers cannot read is a policy that only the team knows, and the team is not the one that breaks.

Pagination at scale is cursor paging, and the reason is simple. Offset paging says skip the first 100 rows, and on a list that changes, the rows shift between pages, so the client skips rows and sees rows twice. Cursor paging says start after this row, and the cursor is stable even when the list changes. Errors at scale are the same story: a code the client can switch on, and the code is stable across versions. A client that switches on an error message breaks the day the message is reworded. A client that switches on a code breaks only when the code changes, and the code changes only in a breaking version.

At 10x the callers, the gateway is the bottleneck, and the rate limit is the design. A GraphQL gateway limits by query cost, not by request count, because one query can ask for a thousand things. A REST gateway limits per caller, per endpoint, and the money endpoints get a stricter limit than the read endpoints. At 100x, the API stops being one API and becomes a platform. Teams own their own services and their own schemas, and the platform owns the gateway, the versioning policy, and the contract tests that fail when a service breaks a shape its callers depend on. The contract test is the part people skip. I write it before the first caller arrives, because a test that fails when the shape breaks is the only thing standing between a rename and a field of broken apps.

openapi: 3.1.0
info:
  title: Orders API
  version: 2.3.0
paths:
  /v2/orders:
    get:
      summary: List orders
      parameters:
        - name: limit
          in: query
          schema: { type: integer, default: 20, maximum: 100 }
        - name: cursor
          in: query
          schema: { type: string }
      responses:
        '200':
          description: A page of orders
          content:
            application/json:
              schema:
                type: object
                properties:
                  orders:
                    type: array
                    items: { $ref: '#/components/schemas/Order' }
                  next_cursor: { type: string, nullable: true }
    post:
      summary: Create an order
      parameters:
        - name: Idempotency-Key
          in: header
          required: true
          schema: { type: string }
      responses:
        '201': { description: Order created }
        '409': { description: Key already used — first answer returned }
components:
  schemas:
    Order:
      type: object
      properties:
        id: { type: string }
        user_name: { type: string }
        userName:
          type: string
          deprecated: true
          description: Use user_name. Removed after old apps retire.
        total: { type: number }
    Error:
      type: object
      required: [code, message]
      properties:
        code: { type: string }
        message: { type: string }

One contract, four promises. The GET lists orders with a limit and a cursor, and the response carries the next cursor, so the client pages by cursor and never skips a row. The POST takes an Idempotency-Key header, and a retry with the same key gets the first answer back with a 409, instead of a second charge. The Order has both user_name and userName, and the old name is marked deprecated, so the old apps keep working while the new clients use the new name. And the Error has a code and a message, and the code is the stable one: clients switch on the code, never on the message, because the message can be reworded and the code cannot change without a breaking version. Read the contract and you can see the whole lesson: page by cursor, retry by key, change by adding, and fail with a code.

One time it broke

This one stayed with me because the deploy was green, the tests were green, and every old app in the field broke at the same time, and nobody on the team knew until the users told us.

What broke

A team shipped a new version of its orders API. The change was small: a field called userName was renamed to user_name, to match the naming style of the rest of the API. The tests passed, because the tests were written against the new name. The deploy went out, because the deploy had no contract test. And the old mobile apps, which could not be updated from the server, kept asking for userName and got nothing back. The field was simply gone from the response, and the apps showed a blank where the name should be. The API was version 2, and the apps were version 2, and nobody had bumped the version, because the team thought a rename was a cleanup, not a break. The error was not a 500. It was a 200 with a missing field, which is worse, because the clients thought the call had worked.

Timeline

  • 09:00the rename ships to production in a routine deploy; the change is one field, and the pull request has two approvals.
  • 09:12the first user report arrives: the orders screen shows a blank name; support assumes it is a bad account and closes the ticket.
  • 09:40the reports multiply; ten users, then fifty, all on the old app version; the on-call checks the API dashboard, all green, because the API is answering 200s.
  • 10:05on-call reads a raw response from the new API and sees it: user_name is there, userName is not.
  • 10:20the team ships both names in the response, with the old one marked deprecated; the old apps work again within minutes.
  • 11:00a contract test is added: it fetches the response with an old client shape and fails if any field the old clients use is missing.
  • Next weekthe breaking-change policy is written down: what counts as breaking, who approves it, and how long the old version lives; a rename is on the list, first.

What changed after

Three things. First, the API ships both names whenever a field is renamed, and the old name is marked deprecated with a date, not removed in the same deploy. A rename is two deploys, not one: the first adds the new name, the second, months later, removes the old one. Second, a contract test runs on every deploy, and it checks the response against the shape the old clients depend on. The test is the thing that would have caught this, because the unit tests only knew the new name. Third, the versioning rule is written down and the team can say it out loud: a rename is a breaking change, a removal is a breaking change, and a breaking change gets a new version, not a quiet edit to the one the old apps already use.

If you remember nothing else

Design exercise

Design the API for a food delivery app. The client needs one screen that shows a restaurant menu, the user's past orders, and a few recommendations. Those live in three services: menu, orders, recommendations. Peak traffic is 5,000 requests per second. Old versions of the mobile app are still out there and they send the v1 request shape. Design the API: which paradigm, how it versions, how it pages, how it fails, and what the gateway does. Say what you would measure.

Clarifying questions to ask

  • Which calls are on the critical path for the screen, and which can load after the screen draws?
  • How old are the old app versions, and how long do you have to keep supporting their v1 shape?
  • Is the traffic steady or spiky, and how long does the peak last?
  • Which endpoints need a login, and which are public, like the menu?
  • Can the client retry a failed order safely, or would a retry place the order twice?

Rough sizing, worked

Start with the numbers from this lesson. One screen needs three things, and three REST calls on a slow connection is three round trips, so I put one GraphQL endpoint in front, or one REST aggregation endpoint at the gateway, and the screen pays one round trip. Peak traffic is 5,000 requests a second, and one gateway instance handles about 1,500 before it queues, so I run 6 instances across two zones, and the gateway pair from Lesson 06 sits in front of them. The menu is paginated by cursor, 20 items a page, and it is cached at the gateway, because the menu is public and it changes slowly. The orders are paginated by cursor too, 20 a page, and they are never cached, because they are the user's own data. The recommendations get a 150 ms timeout with a fallback of an empty list, so a slow recommendations service never blocks the menu and the past orders. The order endpoint takes an idempotency key, because a retry must never place the order twice. Little's Law for the sizing: at 5,000 requests a second, each spending 0.07 seconds inside the API, the number inside the system at once is 5,000 times 0.07, which is 350 concurrent requests, and I size the connection pools for that number plus headroom.

Trade-offs to name

GraphQL buys one round trip and costs query parsing, shaping, and a cost limit, so I limit by query cost, not by request count. REST aggregation buys the same one round trip with a simpler contract, and costs a gateway that stitches answers together, so I keep the aggregation thin: the gateway stitches, it does not decide what to recommend. I keep the v1 to v2 transform in the gateway, so old app versions keep working while the services move on, and I set a date to drop v1 support instead of carrying the transform forever. I check auth at the edge for orders and recommendations, and I let the menu through without a login, because a public menu should not need one. And I set the timeout on every downstream call before I set anything else, because a call with no timeout is a queue that grows until the gateway falls over.

Next in this series
Lesson 126 — eBPF
Coming next