System Design · Lesson 45

Gateway API

Ingress got me into the cluster. It did not get me very far. One resource, one controller, and a pile of annotations that meant different things depending on which controller read them. I moved a route by editing an annotation string and hoping the controller agreed with me. Gateway API is the fix for that, and it is a different shape of fix than I expected. It is not a bigger Ingress. It splits the job three ways. The cluster team picks the controller with a GatewayClass. The platform team owns the Gateway and its listeners. My team owns the HTTPRoute that says which requests go to my service. One request comes in, the Gateway takes it, the HTTPRoute matches it, and the Service hands it to a pod. This lesson follows that request, shows how a header can steer it to a canary, shows how weights split it between two versions, and shows what happens when a route matches the wrong users or no route matches at all. It also shows the part people get wrong. Gateway API is a routing API. It is not a gateway product by itself.

Difficulty: Advanced · Know this first: Lesson 44, Lesson 18
Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in a healthy trip the pod is the real work and the Gateway API hops are nearly free. Switch the scenario and watch the bar change. In the wrong-route scenario, the pod eats almost the whole budget, because a canary sized for ten percent is carrying far more than that. In the no-route scenario, the trip ends at the route match hop, and the bar is short because the failure was fast. A short bar is not always good news. A 26 ms trip that ends in a 404 is still a failed request.

Split the traffic and watch the backends

The budget above is one fixed trip. This box is yours. Move the weight and watch the backends trade the load. I learned this the careful way: a weight is a number in YAML, and a number in YAML can move a whole user base in seconds.

The split of requests between backend A (v1, stable) and backend B (v2, canary). Backend A answers in 50 ms. Backend B answers in 150 ms, because it is new code that has not been tuned yet.

One Gateway instance handles 2000 requests per second before it starts queuing. Past that, the wait lands on the listener hop.

TLS ends at the Gateway. Turning it off makes the trip faster and the traffic readable on the wire.

The Gateway checks routes until one claims the request, so more routes means a slower route match hop.

Backend A (v1)720 requests per second
Backend B (v2)80 requests per second
Pod (weighted)60 ms
Whole trip~120 ms

Assumptions, stated plainly: backend A answers in 50 ms, backend B in 150 ms, and the pod hop is their weighted average. One Gateway instance handles 2000 requests per second before queuing. The fixed hops total 47 ms: Gateway 4, header routing 6, policy 10, split 3, Service 12, answer 12.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Go deeper

The lesson above is one Gateway, one route, and one canary. Production is the rest of it. Here is what I actually spend my time on once Gateway API is running.

What Gateway API is not

This is the part people skip, and it costs them a migration. Gateway API is a routing API. It standardises how traffic gets matched and sent to backends. It does not ship auth, rate limiting, request transformation, a web application firewall, a developer portal, or billing. Those come from the implementation you pick, or from policy objects attached to the Gateway or the route. When someone says they are replacing their API gateway with Gateway API, I ask what they mean. If they mean the routing rules, yes. If they mean the whole product, no. The product still exists. It just reads Gateway API objects now instead of its own config format. Lesson 18 covered what a gateway does. This lesson covers how Kubernetes standardises the routing half of it.

The role split is the real change

Ingress gave every team the same object with the same annotations. Gateway API gives each role its own object. The cluster team owns the GatewayClass and picks the controller. The platform team owns the Gateway, the listeners, and the certificates. The app team owns the HTTPRoute, and only the HTTPRoute. A bad edit by my team can break my route. It cannot delete the listener. That boundary is worth more than any new matching feature. It is also what makes the migration safe to do one route at a time. I move one HTTPRoute, watch it, then move the next. The old Ingress keeps serving the routes I have not moved yet.

East-west gateways, inside the cluster

The Gateway in this lesson faces the internet. The same API works inside the cluster, between services. A mesh Gateway takes service-to-service traffic, applies the same matching and splitting, and lets one service canary another service without either team changing code. I use it when a service call is load bearing enough to deserve its own route rules. I do not use it for every call. A Gateway per service pair is a pile of objects nobody can hold in their head. I start with the calls that have actually caused incidents.

mTLS at the Gateway

TLS termination at the listener is the common case, and it is what this lesson shows. Two other modes matter. TLS passthrough sends the encrypted traffic straight to the backend, and the Gateway never sees inside it. That fits when the backend owns its own certificate and the Gateway should not terminate. Mutual TLS goes further: the Gateway checks the client certificate too, so only known clients get in. That fits service-to-service traffic and partner traffic, not public web traffic. And the hop after the Gateway can be encrypted too. A BackendTLSPolicy tells the Gateway to use TLS when it talks to the backend Service, so the inside traffic is not plain text either. I turn that on for traffic that crosses a trust boundary inside the cluster. I leave it off for traffic that never leaves a private network, because the extra handshake buys nothing there.

Policy attachment patterns

Policies attach by reference, and the reference says what the policy targets. A policy attached to a Gateway applies to every route on it. A policy attached to an HTTPRoute applies to that route only. A policy attached to a Service applies to that backend. When two policies overlap, the most specific one wins for its target, and the Gateway policy is the fallback. I attach the defaults at the Gateway, auth and the base rate limit, and I attach the exceptions at the route, a stricter limit on the checkout route, a looser one on the menu route. The rule I follow is simple. If every route needs it, it goes on the Gateway. If one route needs it, it goes on the route. And I check the policy status after every attach. A policy the controller does not understand is ignored without an error, and an ignored rate limit looks exactly like no rate limit until the traffic arrives.

What changes at 10x

At ten times the routes, the route table stops being a file I can read in one sitting. I split routes by team and by namespace, one HTTPRoute per team per hostname, and I lint the matches in CI before they apply. A conflicting match is caught by a test, not by a user. At ten times the traffic, the Gateway replicas go up, and the per-instance queue from the simulator becomes the number I watch. I also start caring about how many routes one Gateway holds, because every route is matching work on every request. The simulator's route-count slider is that lesson in miniature.

What changes at 100x

At a hundred times the traffic, one Gateway in one cluster is not enough. I run a Gateway in each cluster, put the same hostname on all of them, and steer traffic with DNS or a global load balancer. The hard part is not the steering. It is keeping the routes identical across clusters. A route that exists in one cluster and not the other is a user who gets a 404 based on which cluster answered. So the routes live in one repo, apply to every cluster from the same commit, and a drift check pages me when a cluster falls behind. At this scale, the Gateway API objects are the contract between teams, and the contract is enforced by CI, not by memory.

Gateway and HTTPRoute YAML I actually use

Short snippet, shown in a code block. This is a standard Gateway API example, not tied to any one vendor.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: shop-gateway
  namespace: ingress
spec:
  gatewayClassName: envoy
  listeners:
    - name: https
      port: 443
      protocol: HTTPS
      hostname: shop.example.com
      tls:
        mode: Terminate
        certificateRefs:
          - name: shop-tls
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: checkout-route
  namespace: shop
spec:
  parentRefs:
    - name: shop-gateway
      namespace: ingress
  hostnames:
    - shop.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /checkout
          headers:
            - name: x-beta-user
              value: "true"
      filters:
        - type: RequestHeaderModifier
          requestHeaderModifier:
            add:
              - name: x-route-version
                value: canary
      backendRefs:
        - name: checkout-v2
          port: 8080
          weight: 10
        - name: checkout-v1
          port: 8080
          weight: 90
    - matches:
        - path:
            type: PathPrefix
            value: /checkout
      backendRefs:
        - name: checkout-v1
          port: 8080
          weight: 100

What the snippet means: the Gateway listens on 443 for shop.example.com and ends TLS with the shop-tls certificate. The HTTPRoute attaches to that Gateway from the shop namespace, so it can live with the team that owns checkout. The first rule matches beta users by header and splits them 10 to v2 and 90 to v1. The second rule is the default, and it sends everyone else to v1. The header filter tags the canary requests so the backend and the logs know which path they took. And there is no timeout in this route, because the standard API does not define one. My timeout lives in an attached policy, where I can see it and review it, not hidden in a route field that does not exist.

One time it broke

This one stayed with me because nothing failed. No pod crashed, no certificate expired, no Gateway went down. The new route simply never took effect, and we shipped around a route that was not there.

What broke

Two teams shared one hostname. The search team owned the routes for /search. The checkout team owned the routes for /checkout. Each team wrote its own HTTPRoute in its own namespace, and both attached to the same Gateway. That is a supported setup, and it worked for months. Then the checkout team added a second HTTPRoute for /checkout, in a new namespace, for a rebuilt checkout service. The new route was valid YAML. It applied cleanly. It did nothing. The old route was older, and when two routes claim the same hostname and path, the oldest one wins. The new route sat there, accepted on paper and ignored in practice, while all the traffic kept going to the old backend. The team tested their new service, saw the old responses, and concluded their deploy had failed. They redeployed twice. Nothing changed, because the deploy was never the problem. The route conflict was.

Timeline

  • 10:00The checkout team applies the new HTTPRoute for /checkout in the new namespace. The apply succeeds. No error, no warning.
  • 10:15The team tests the new checkout service through the public hostname. They get the old responses. They assume the new pods are not serving yet.
  • 10:40The team redeploys the new service and checks the pod logs. The pods are healthy. They have received zero requests.
  • 11:05I check the Gateway access log. Every /checkout request is going to the old backend. The new backend has no traffic at all.
  • 11:20I list the HTTPRoutes for the hostname. There are two claiming /checkout. The old one is older, so it wins. The new one is attached and ignored. The status on the new route says it is accepted, which is true and useless. Accepted means the object is valid. It does not mean it wins the conflict.
  • 11:35I delete the old route, and the new route takes over within seconds. All the traffic moves to the new backend at once, with no canary and no weight split, because the new route was written as a full cutover. The new backend holds, but only just. The team had planned a 10 percent canary. They got 100 percent in one move.
  • 12:00I put the old route back, set the new route to weight 10 against the old backend, and let the canary run the way it was planned.

What changed after

Three things, and I still do all three. First, one route owner per hostname and path. Two routes are allowed to claim different paths on the same hostname, but never the same path. That rule lives in a CI check now, not in a wiki page. The check lists every HTTPRoute for a hostname and fails the change if two of them claim the same path. Second, I added a status check to the deploy pipeline. Accepted is not enough. The pipeline now confirms the route is the one actually serving the path, by reading the Gateway's effective route list, before the deploy is called done. Third, I changed how we cut over. A new route never goes in at weight 100 on its first apply. It goes in at weight 10 with the old backend still listed, and the weight moves up in steps. A route that takes over by accident should take over 10 percent of the traffic, not all of it.

If you remember nothing else

Try it yourself

This is the kind of question I would ask in an interview, and the kind I ask myself before I move a real route. Take a minute with it before you read my approach.

The prompt

You run a shop with three teams in one cluster. Checkout, search, and accounts each own their own services and deploy on their own schedule. They share one public hostname, shop.example.com. Checkout needs canary deploys. Search needs a header rule that sends internal test traffic to a staging backend. Accounts needs mTLS for its partner traffic. Design the Gateway API setup. Who owns which object, how do the routes avoid conflicting, and what happens when checkout moves a weight?

Clarifying questions to ask first

I would ask five things before I drew anything. Which team owns the Gateway and the listeners, and which teams only own routes? How many routes will there be in a year, and who reviews a route change? Does the header rule for internal test traffic use a header the public internet can also set, or a header only the internal client sets? Which traffic needs mTLS, the public traffic or only the partner traffic, and where does that check happen? And what is the rollback for a bad weight change: who can change it, and how fast does the old split come back?

Rough sizing, worked

Three teams, one hostname, one Gateway with two listeners: 443 for public HTTPS and 8443 for partner mTLS. Each team owns one HTTPRoute per path prefix in its own namespace, so three routes to start, growing to about ten as new paths appear. The Gateway runs 3 replicas across two zones, sized for a peak of 4,000 requests per second against a per-instance capacity of 2,000, so one replica dying still leaves headroom. Checkout canary starts at weight 10 to v2, moves to 50 after an hour of clean error rates, then to 100. The weights live in the checkout team's HTTPRoute only, so a weight change is a one-object apply that takes effect in seconds, and the rollback is the same apply in reverse. The search header rule matches on a header only the internal test client sets, and it is scoped to a path prefix the public app never calls, so a public user cannot stumble into the staging backend.

Trade-offs to name

One Gateway for all three teams is simpler to run and a bigger blast radius. A bad listener change hits everyone. I accept that, because the listeners change rarely and the routes change often, and the routes are split by team. The alternative, one Gateway per team, is safer per team and harder to run, with three hostnames or three ports to explain to clients. I keep the shared Gateway and put the safety in CI: no two routes may claim the same hostname and path, and every route change is reviewed by the team that owns the path. I put mTLS on its own listener, not on the public one, because a client certificate check on public traffic would lock out every real user. And I do not put the staging backend behind the same route as production. The header rule points at a separate backend, so a bad header match sends test traffic to staging, not production traffic to a test service.

Next in this series
Lesson 46 — Networking Deep Dive
Coming next