Platform Engineering · Lesson 44

Kubernetes Ingress

A Service inside Kubernetes has an address, but only inside the cluster. Nobody outside can reach it. That is the design. So how does a real user on the internet reach a pod? The answer is a small chain of pieces that each do one job, and the chain only works when all of them agree. The Ingress resource says which host and path goes where. The ingress controller reads that resource and builds the real routing rules. A cloud load balancer hands traffic to the controller. The Service hands it to a pod. This lesson follows one request from a user outside the cluster, through all four pieces, to a pod and back. It also shows the three ways I have seen this chain break in production: a wrong ingress class, a path that matched the wrong backend, and an annotation with a typo that the controller silently ignored.

Difficulty: Advanced · Know this first: Lesson 18, Lesson 06
Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in a healthy trip the pod does the real work and the Ingress hops are nearly free. Switch the scenario and watch the bar change. In the wrong ingress class scenario, the trip is fast because almost nothing happens. In the wrong path scenario, the trip is slower than normal because the wrong pod does real work before it says no.

Route it yourself and watch the split

The budget above is three fixed stories. This box is yours. Move the sliders and watch what the Ingress does when the traffic grows, when the canary takes more of it, and when TLS goes away. I learned this the slow way: the Ingress is not magic. It is a controller with a capacity, and when the traffic passes that capacity, the queue is where your users wait.

How much traffic arrives. One controller pod handles about 500 requests per second before it starts queuing.

How much traffic goes to the canary backend. The canary answers in 70 ms, the stable backend in 45 ms.

Turn TLS off and the trip gets 22 ms faster on paper. The traffic travels in plain text.

How the load spreads. More pods means less load on each pod, and less queue at the controller.

Traffic split
Per-pod load
Hop costs
Whole trip

The default total is 126 ms, 3 ms above the normal trip, because 10% of traffic runs on the slower canary backend. The simulator works on the normal scenario: moving a control switches the diagram back to it.

Ingress, from scratch: the door and the thing that builds the door

Extra diagram. I read the picture first: the boxes are the system, the moving dot is a request, and each step adds one piece until the full picture is there. No text needed to follow it.

The Ingress resource alone does nothingOne controller pod handles about 500 rpsTLS termination costs 22 ms
123456789

Outside clusterControllerInside clusterMoving dots: request · response

The reload trip: change one line, move the traffic

Extra diagram. I read the picture first: the boxes are the system, the moving dot is an Ingress change, and each step adds one piece until the full picture is there. No text needed to follow it.

Reload makes new rules liveWrong class means the rules are ignoredTest every route from outside the cluster
1234567

Happy pathReloadFailure modesMoving dots: request · response

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

The lesson above follows one request from the internet to a pod. This section covers the same chain four times, from the first route to the level where Ingress runs across teams without breaking them.

Basic — One door into the cluster

Definition: A Service inside Kubernetes has an address, but only inside the cluster. Nobody outside can reach it. That is the design. The Ingress is the door. It says which host and path goes where, and the ingress controller builds the real routing rules from it.

Types:

  • The Ingress resource — host and path, written in YAML.
  • The controller — the piece that reads the resource and configures the routing.
  • The load balancer — the cloud piece that hands traffic to the controller.
  • The Service — the last hop, handing traffic to a pod.

Uses: I use this level the first time I expose a service. One host, one path, one backend. I test it from outside the cluster, because a route that works from inside and fails from outside is the most common surprise.

Dimensions:

  • Load: At one service, one Ingress is a few lines of YAML. At fifty services, the controller builds one config from all of them, and one bad rule can break another team's routing.
  • Breaks: It breaks on one word in a YAML file. A regex path, a missing slash, a rewrite that sends the request somewhere the pod does not expect. I notice it when the dashboards are green and the users get 404s.
  • Cost: The cost is a controller to run and a load balancer to pay for. The Ingress itself is cheap. The load balancer is the line on the bill.
  • Trade-off: I give up direct access to get one door. Every outside request passes through the same chain. That is slower to set up and far easier to reason about.

Intermediate — Host, path, and TLS

Definition: Routing has three questions: which host, which path, and is the connection encrypted. The Ingress answers all three. The host picks the service. The path picks the handler. TLS ends at the controller, so the pod sees plain traffic and the user sees a lock.

Types:

  • Host routing — one hostname per service, or one hostname split by path.
  • Path matching — exact, prefix, or regex. Regex is the one that bites.
  • TLS termination — the certificate lives on the Ingress, not on every pod.
  • The rewrite — changing the path before the pod sees it, so the pod gets clean paths.

Uses: I use this level when one hostname serves two services, or when the API team wants clean paths. I write the path rule, I test it with a real request, and I check the pod logs to see what path actually arrived.

Dimensions:

  • Load: At low traffic, TLS termination costs nothing noticeable. At high traffic, the controller does the encryption work for every request, and it needs enough CPU or it becomes the bottleneck.
  • Breaks: It breaks when a certificate expires and nobody notices, or when a rewrite sends /api/v1/orders to the pod as /orders and the pod has no such route. I notice it in the 404 rate, not in the pod logs.
  • Cost: The cost is certificate management. A certificate that renews itself is free. One that needs a human every ninety days is a page waiting to happen.
  • Trade-off: I give up end-to-end encryption to get simple routing. TLS ends at the controller. If I need encryption all the way to the pod, I need the next level up, and the next lesson.

Advanced — Controllers, annotations, and canary

Definition: The Ingress resource is simple, and simple is the point. It does host and path routing, TLS, and a few annotations. The day I need header matching, traffic splitting by weight, or a retry policy, the Ingress starts to strain. Annotations are where that strain shows: one string, different meanings on different controllers.

Types:

  • Annotations — controller-specific settings, written as strings, easy to get wrong.
  • Canary by weight — a small share of traffic to the new version, watched before the rest follows.
  • One controller per team — isolation, so one team's bad rule cannot break another's routing.
  • The migration signal — when routing needs outgrow annotations, I look at Gateway API.

Uses: I use this level when a deploy needs a canary, or when two teams share a controller and start stepping on each other. I split the controller when the blast radius gets too big, not before.

Dimensions:

  • Load: At one controller, the config is one file built from every Ingress in the cluster. At hundreds of Ingresses, a bad rule in one team's file rebuilds the config for everyone, and the reload can drop traffic for a second.
  • Breaks: It breaks when an annotation means different things on different controllers. I moved a route by editing an annotation string and hoping the controller agreed with me. I notice it when the route works in staging and does nothing in production.
  • Cost: The cost is the controller per team, and the review time for every Ingress change. Each extra controller is another thing to upgrade and another bill.
  • Trade-off: I give up the simplicity of one shared controller to get isolation. Shared is cheap and the blast radius is the whole cluster. Split is expensive and the blast stays small.

Master — Ingress across teams at scale

Definition: At this level Ingress is a platform concern, not a per-service file. The platform team owns the controllers, the certificate pipeline, and the rules about what an Ingress may say. Teams write routes. The platform decides how routes become reality.

Types:

  • The controller fleet — shared by default, split by team or environment when the blast radius demands it.
  • Certificate automation — certificates issued and renewed by a controller, never by a human.
  • Policy on Ingress — no route without an owner, no wildcard host without a review.
  • The migration path — Ingress where it fits, Gateway API where the routing needs more, both running at once.

Uses: I use this level when the cluster has dozens of teams. The rule is simple: a team can break its own routing, never another team's. I enforce that with separate controllers and with policy, not with a request in a chat channel.

Dimensions:

  • Load: At this scale the controller is a service with its own SLO. A reload storm from hundreds of Ingress changes can drop traffic. I rate-limit changes and I watch the reload time.
  • Breaks: It breaks when the platform team is the bottleneck. Every route change waits on a ticket. I notice it when teams start working around the Ingress instead of through it.
  • Cost: The cost is a platform team, controller capacity, and the certificate pipeline. It is real money and real headcount, and it is cheaper than fifty teams each solving routing alone.
  • Trade-off: I give up team freedom to get a door that holds. Teams cannot do anything they want with routing. In return, nobody's bad YAML takes down another team's users.

Go deeper

The lesson above is the happy path with two config failures. Production is the rest of it. Here is what I actually spend my time on once the Ingress is running.

Trade-offs I weigh every time

The Ingress is simple, and simple is the point. It does host and path routing, TLS, and a few annotations. The day I need header matching, traffic splitting by weight, or a retry policy, the Ingress starts to strain. That is when I look at the Gateway API, which is the next lesson. I do not jump to it early. The Ingress is simpler to read, simpler to review, and every controller supports it. I move when the routing needs outgrow what annotations can express, not before. The other trade-off is how many controllers I run. One shared controller for the whole cluster is simple and cheap. It is also a shared blast radius. A bad rule in one team's Ingress can break another team's routing, because the controller builds one config from all of them. Separate controllers per team or per environment cost more and isolate the blast. I start shared and split when the teams start stepping on each other.

Canary routing with the Ingress

The canary in the simulator is not a theory. It is three annotations on a second Ingress. The canary Ingress names the same host, points at the canary Service, and sets nginx.ingress.kubernetes.io/canary to "true" and nginx.ingress.kubernetes.io/canary-weight to "10". The controller sends 10 percent of requests to the canary Service and the rest to the stable one. I watch the canary error rate for an hour before I raise the weight. The rule I never break is that the canary and the stable version answer the same contract. A canary that returns a different shape is not a test. It is a bug that only 10 percent of users see.

Rate limiting at the edge

The Ingress is the cheapest place to limit traffic, because a rejected request never reaches a pod. The NGINX annotations are nginx.ingress.kubernetes.io/limit-rps for requests per second and nginx.ingress.kubernetes.io/limit-connections for open connections per client. I set the limit per client IP or per host, and I set it higher than the real peak, so normal traffic never notices it. The limit is a fence, not a throttle. If it fires during normal traffic, the number is wrong, not the traffic. And I count it in one place. A limit that each controller pod counts on its own lets more traffic through every time I add a replica.

external-dns, or the DNS that writes itself

Every host in an Ingress needs a DNS record, and I used to create those by hand. Then I forgot one, and the Ingress was perfect while the name did not resolve. external-dns fixes that. It watches the Ingress resources and creates the DNS records from the hosts it finds. I add a host to the Ingress and the record appears. I delete the Ingress and the record goes away. The catch is the same as every automation: it does exactly what the Ingress says. A typo in the host creates a real DNS record for a name nobody wanted, so I still review the host list on every change.

What changes at 10x

At ten times the traffic, the controller stops being a box I think about once a year and becomes a capacity number I track every week. I watch requests per second per controller pod, not just the total, and I scale the controller replicas before the queue forms. I also stop letting every team write annotations freely. Annotations are code that runs on every request, and a bad regex in a rewrite annotation can slow the whole controller. So annotation changes get reviewed like code, and the rewrite rules get tested in staging with real paths.

What changes at 100x

At a hundred times the traffic, one Ingress and one controller are not enough. I split the controllers by environment or by team, so one bad rule cannot take down every host at once. The certificate count grows too, and I move to a wildcard certificate per domain instead of one certificate per host, because cert-manager renewing hundreds of certificates is a job in itself. And the routing moves to the Gateway API, because at this scale I need header matching and weight-based splits that annotations cannot express cleanly. The Ingress does not disappear. It stays for the simple hosts, and the Gateway API takes the complicated ones.

Ingress YAML I actually use

This is a generic NGINX Ingress with the annotations from this lesson:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop
  namespace: shop
  annotations:
    nginx.ingress.kubernetes.io/use-regex: "true"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/limit-rps: "20"
    nginx.ingress.kubernetes.io/proxy-body-size: 10m
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - shop.example.com
    secretName: shop-tls
  rules:
  - host: shop.example.com
    http:
      paths:
      - path: /api(/|$)(.*)
        pathType: ImplementationSpecific
        backend:
          service:
            name: api
            port:
              number: 80
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web
            port:
              number: 80

The three lines that matter most are the first three annotations and the pathType on the API rule. use-regex is "true" because the API path is a regex, and a regex path needs ImplementationSpecific, not Prefix. Without those two, NGINX treats the path as a literal string and nothing matches. rewrite-target: /$2 strips the /api prefix, so the API pod sees /orders instead of /api/orders. limit-rps: "20" caps each client at 20 requests per second before a pod ever sees the traffic. ingressClassName: nginx is the line that decides whether any of this happens at all. If the controller watches a different class, this whole Ingress is ignored.

One time it broke

This one stayed with me because every dashboard was green and every user was getting a 404. The pods were fine. The Service was fine. The problem was one word in a YAML file.

What broke

A deploy changed the Ingress for an API service. The team wanted the API pods to see clean paths, so they added a rewrite: requests to /api/v1/orders should reach the API pod as /orders. The path in the Ingress was written as a regex, /api/v1/(.*), with a rewrite target of /$1. The change looked right. It went out on a Friday afternoon. From that moment, every request to the API returned a 404 from the default backend. Not from the app. From the default backend, which answers for hosts and paths it knows nothing about. The app pods were healthy. The Service had four endpoints. The controller was running. Nothing was down, and nothing worked.

Timeline

  • 16:10The Ingress change deploys. It looks clean. No errors in the deploy log, and the first few minutes are quiet because traffic is low.
  • 16:14The first 404 reports come in. The app dashboard is green. The pods are green. That mismatch tells me the problem is in front of the app.
  • 16:22I check the controller access log. The 404s are coming from the default backend, not from the API Service. The request never reached the API pods at all.
  • 16:30I check the Service endpoints. Four endpoints, all ready. The Service is fine, which means the routing never chose it.
  • 16:41I read the Ingress with kubectl describe ingress and compare it with the previous version. The path is a regex, but the pathType is Prefix, and there is no use-regex annotation. NGINX treated the regex as a literal string, so no real request ever matched it.
  • 16:52I fix it: use-regex set to "true", pathType changed to ImplementationSpecific. The controller reloads.
  • 16:55The 404s stop. Requests reach the API pods again, with the prefix stripped, exactly as intended.
  • 17:15I confirm the fix from three networks and check the default backend 404 rate is back to zero.

What changed after

Three things, and I still do all three. First, Ingress changes get reviewed like code. A second person reads the path, the pathType, and the annotations before it deploys, and a regex path without use-regex is a review failure, not a deploy failure. Second, I added an alert on the default backend 404 rate. A spike in default backend 404s is now a page, because it means routing is broken, not the app. Third, I added a staging check that hits every real path after an Ingress change, not just the homepage. A deploy that only tests / will never catch a broken /api rule, and that is the rule that matters.

If you remember nothing else

Try it yourself

This is the kind of question I would ask in an interview, and the kind I ask myself before I change a real Ingress. Take a minute with it before you read my approach.

The prompt

You run a multi-tenant app. There are 200 tenants, and each one gets its own host: tenant1.app.example.com, tenant2.app.example.com, and so on. Each tenant needs TLS. A new version of the checkout page is ready, and you want to canary it to 10 percent of traffic. One tenant is a heavy user and must never be able to take down the others. Design the Ingress for this app. How do you handle 200 hosts and 200 certificates, where does the canary live, and how do you stop one tenant from hurting the rest?

Clarifying questions to ask first

I would ask five things before I sized anything. Do the tenants share one app, or does each tenant get its own pods? How strict is the isolation: can tenants share a controller, or must a bad rule in one tenant never affect another? How fresh must the canary decision be: can I change the weight in minutes, or does it need to be seconds? What does the heavy tenant actually do: many requests, big payloads, or slow queries? And what happens when a tenant's certificate fails to renew: does that tenant go dark, or does it fall back to plain HTTP?

Rough sizing, worked

200 hosts and 200 certificates is a job for a wildcard, not 200 separate certificates. One wildcard certificate for *.app.example.com covers every tenant, and cert-manager renews one certificate instead of 200. The Ingress gets one rule per tenant host, or one rule with a wildcard host if the app routes tenants by host header itself. I prefer one rule per tenant in the Ingress, because the rule list is the audit trail of who exists. The controller runs as 3 replicas behind the cloud load balancer. At 300 requests per second total, that is 100 per replica, well under the 500 per pod queue point from the simulator. The canary is a second Ingress with the same host pattern, the canary annotations set, and a weight of 10. The heavy tenant gets a rate limit annotation on its own Ingress rule, 20 requests per second, so its burst stops at the controller and never reaches the shared pods.

Trade-offs to name

A wildcard certificate buys simplicity and costs blast radius: if it expires, all 200 tenants go dark at once, so the expiry alert pages at 30, 14, and 7 days. One shared controller buys simplicity and costs isolation: a bad annotation in one tenant's Ingress can slow the controller for all of them, so annotation changes get reviewed like code. Separate controllers per tenant would buy isolation and cost 200 controllers, which nobody wants to run. I rate limit per tenant at the Ingress, because a rejected request never reaches a pod, but I also keep a per-tenant limit inside the app, because the Ingress limit counts requests and the app limit counts work, and those are different things. And I say the canary out loud: 10 percent of traffic on the new checkout is a test, not a release, and the weight only goes up when the canary error rate stays flat for an hour.

Next in this series
Lesson 45 — Gateway API