A Service inside Kubernetes has an address, but only inside the cluster. Nobody outside can reach it. That is the design. So how does a real user on the internet reach a pod? The answer is a small chain of pieces that each do one job, and the chain only works when all of them agree. The Ingress resource says which host and path goes where. The ingress controller reads that resource and builds the real routing rules. A cloud load balancer hands traffic to the controller. The Service hands it to a pod. This lesson follows one request from a user outside the cluster, through all four pieces, to a pod and back. It also shows the three ways I have seen this chain break in production: a wrong ingress class, a path that matched the wrong backend, and an annotation with a typo that the controller silently ignored.
Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: in a healthy trip the pod does the real work and the Ingress hops are nearly free. Switch the scenario and watch the bar change. In the wrong ingress class scenario, the trip is fast because almost nothing happens. In the wrong path scenario, the trip is slower than normal because the wrong pod does real work before it says no.
The budget above is three fixed stories. This box is yours. Move the sliders and watch what the Ingress does when the traffic grows, when the canary takes more of it, and when TLS goes away. I learned this the slow way: the Ingress is not magic. It is a controller with a capacity, and when the traffic passes that capacity, the queue is where your users wait.
How much traffic arrives. One controller pod handles about 500 requests per second before it starts queuing.
How much traffic goes to the canary backend. The canary answers in 70 ms, the stable backend in 45 ms.
Turn TLS off and the trip gets 22 ms faster on paper. The traffic travels in plain text.
How the load spreads. More pods means less load on each pod, and less queue at the controller.
The default total is 126 ms, 3 ms above the normal trip, because 10% of traffic runs on the slower canary backend. The simulator works on the normal scenario: moving a control switches the diagram back to it.
Extra diagram. I read the picture first: the boxes are the system, the moving dot is a request, and each step adds one piece until the full picture is there. No text needed to follow it.
Extra diagram. I read the picture first: the boxes are the system, the moving dot is an Ingress change, and each step adds one piece until the full picture is there. No text needed to follow it.
Three questions. No score is saved anywhere — this is just you against the material.
The lesson above follows one request from the internet to a pod. This section covers the same chain four times, from the first route to the level where Ingress runs across teams without breaking them.
Definition: A Service inside Kubernetes has an address, but only inside the cluster. Nobody outside can reach it. That is the design. The Ingress is the door. It says which host and path goes where, and the ingress controller builds the real routing rules from it.
Types:
Uses: I use this level the first time I expose a service. One host, one path, one backend. I test it from outside the cluster, because a route that works from inside and fails from outside is the most common surprise.
Dimensions:
Definition: Routing has three questions: which host, which path, and is the connection encrypted. The Ingress answers all three. The host picks the service. The path picks the handler. TLS ends at the controller, so the pod sees plain traffic and the user sees a lock.
Types:
Uses: I use this level when one hostname serves two services, or when the API team wants clean paths. I write the path rule, I test it with a real request, and I check the pod logs to see what path actually arrived.
Dimensions:
Definition: The Ingress resource is simple, and simple is the point. It does host and path routing, TLS, and a few annotations. The day I need header matching, traffic splitting by weight, or a retry policy, the Ingress starts to strain. Annotations are where that strain shows: one string, different meanings on different controllers.
Types:
Uses: I use this level when a deploy needs a canary, or when two teams share a controller and start stepping on each other. I split the controller when the blast radius gets too big, not before.
Dimensions:
Definition: At this level Ingress is a platform concern, not a per-service file. The platform team owns the controllers, the certificate pipeline, and the rules about what an Ingress may say. Teams write routes. The platform decides how routes become reality.
Types:
Uses: I use this level when the cluster has dozens of teams. The rule is simple: a team can break its own routing, never another team's. I enforce that with separate controllers and with policy, not with a request in a chat channel.
Dimensions:
The lesson above is the happy path with two config failures. Production is the rest of it. Here is what I actually spend my time on once the Ingress is running.
The Ingress is simple, and simple is the point. It does host and path routing, TLS, and a few annotations. The day I need header matching, traffic splitting by weight, or a retry policy, the Ingress starts to strain. That is when I look at the Gateway API, which is the next lesson. I do not jump to it early. The Ingress is simpler to read, simpler to review, and every controller supports it. I move when the routing needs outgrow what annotations can express, not before. The other trade-off is how many controllers I run. One shared controller for the whole cluster is simple and cheap. It is also a shared blast radius. A bad rule in one team's Ingress can break another team's routing, because the controller builds one config from all of them. Separate controllers per team or per environment cost more and isolate the blast. I start shared and split when the teams start stepping on each other.
The canary in the simulator is not a theory. It is three annotations on a second Ingress. The canary Ingress names the same host, points at the canary Service, and sets nginx.ingress.kubernetes.io/canary to "true" and nginx.ingress.kubernetes.io/canary-weight to "10". The controller sends 10 percent of requests to the canary Service and the rest to the stable one. I watch the canary error rate for an hour before I raise the weight. The rule I never break is that the canary and the stable version answer the same contract. A canary that returns a different shape is not a test. It is a bug that only 10 percent of users see.
The Ingress is the cheapest place to limit traffic, because a rejected request never reaches a pod. The NGINX annotations are nginx.ingress.kubernetes.io/limit-rps for requests per second and nginx.ingress.kubernetes.io/limit-connections for open connections per client. I set the limit per client IP or per host, and I set it higher than the real peak, so normal traffic never notices it. The limit is a fence, not a throttle. If it fires during normal traffic, the number is wrong, not the traffic. And I count it in one place. A limit that each controller pod counts on its own lets more traffic through every time I add a replica.
Every host in an Ingress needs a DNS record, and I used to create those by hand. Then I forgot one, and the Ingress was perfect while the name did not resolve. external-dns fixes that. It watches the Ingress resources and creates the DNS records from the hosts it finds. I add a host to the Ingress and the record appears. I delete the Ingress and the record goes away. The catch is the same as every automation: it does exactly what the Ingress says. A typo in the host creates a real DNS record for a name nobody wanted, so I still review the host list on every change.
At ten times the traffic, the controller stops being a box I think about once a year and becomes a capacity number I track every week. I watch requests per second per controller pod, not just the total, and I scale the controller replicas before the queue forms. I also stop letting every team write annotations freely. Annotations are code that runs on every request, and a bad regex in a rewrite annotation can slow the whole controller. So annotation changes get reviewed like code, and the rewrite rules get tested in staging with real paths.
At a hundred times the traffic, one Ingress and one controller are not enough. I split the controllers by environment or by team, so one bad rule cannot take down every host at once. The certificate count grows too, and I move to a wildcard certificate per domain instead of one certificate per host, because cert-manager renewing hundreds of certificates is a job in itself. And the routing moves to the Gateway API, because at this scale I need header matching and weight-based splits that annotations cannot express cleanly. The Ingress does not disappear. It stays for the simple hosts, and the Gateway API takes the complicated ones.
This is a generic NGINX Ingress with the annotations from this lesson:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: shop
namespace: shop
annotations:
nginx.ingress.kubernetes.io/use-regex: "true"
nginx.ingress.kubernetes.io/rewrite-target: /$2
nginx.ingress.kubernetes.io/limit-rps: "20"
nginx.ingress.kubernetes.io/proxy-body-size: 10m
cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
ingressClassName: nginx
tls:
- hosts:
- shop.example.com
secretName: shop-tls
rules:
- host: shop.example.com
http:
paths:
- path: /api(/|$)(.*)
pathType: ImplementationSpecific
backend:
service:
name: api
port:
number: 80
- path: /
pathType: Prefix
backend:
service:
name: web
port:
number: 80
The three lines that matter most are the first three annotations and the pathType on the API rule. use-regex is "true" because the API path is a regex, and a regex path needs ImplementationSpecific, not Prefix. Without those two, NGINX treats the path as a literal string and nothing matches. rewrite-target: /$2 strips the /api prefix, so the API pod sees /orders instead of /api/orders. limit-rps: "20" caps each client at 20 requests per second before a pod ever sees the traffic. ingressClassName: nginx is the line that decides whether any of this happens at all. If the controller watches a different class, this whole Ingress is ignored.
This one stayed with me because every dashboard was green and every user was getting a 404. The pods were fine. The Service was fine. The problem was one word in a YAML file.
A deploy changed the Ingress for an API service. The team wanted the API pods to see clean paths, so they added a rewrite: requests to /api/v1/orders should reach the API pod as /orders. The path in the Ingress was written as a regex, /api/v1/(.*), with a rewrite target of /$1. The change looked right. It went out on a Friday afternoon. From that moment, every request to the API returned a 404 from the default backend. Not from the app. From the default backend, which answers for hosts and paths it knows nothing about. The app pods were healthy. The Service had four endpoints. The controller was running. Nothing was down, and nothing worked.
kubectl describe ingress and compare it with the previous version. The path is a regex, but the pathType is Prefix, and there is no use-regex annotation. NGINX treated the regex as a literal string, so no real request ever matched it.use-regex set to "true", pathType changed to ImplementationSpecific. The controller reloads.Three things, and I still do all three. First, Ingress changes get reviewed like code. A second person reads the path, the pathType, and the annotations before it deploys, and a regex path without use-regex is a review failure, not a deploy failure. Second, I added an alert on the default backend 404 rate. A spike in default backend 404s is now a page, because it means routing is broken, not the app. Third, I added a staging check that hits every real path after an Ingress change, not just the homepage. A deploy that only tests / will never catch a broken /api rule, and that is the rule that matters.
/api catches /api-docs, and an Exact rule for /app misses /app/dashboard. Read the pathType on every rule.This is the kind of question I would ask in an interview, and the kind I ask myself before I change a real Ingress. Take a minute with it before you read my approach.
You run a multi-tenant app. There are 200 tenants, and each one gets its own host: tenant1.app.example.com, tenant2.app.example.com, and so on. Each tenant needs TLS. A new version of the checkout page is ready, and you want to canary it to 10 percent of traffic. One tenant is a heavy user and must never be able to take down the others. Design the Ingress for this app. How do you handle 200 hosts and 200 certificates, where does the canary live, and how do you stop one tenant from hurting the rest?
I would ask five things before I sized anything. Do the tenants share one app, or does each tenant get its own pods? How strict is the isolation: can tenants share a controller, or must a bad rule in one tenant never affect another? How fresh must the canary decision be: can I change the weight in minutes, or does it need to be seconds? What does the heavy tenant actually do: many requests, big payloads, or slow queries? And what happens when a tenant's certificate fails to renew: does that tenant go dark, or does it fall back to plain HTTP?
200 hosts and 200 certificates is a job for a wildcard, not 200 separate certificates. One wildcard certificate for *.app.example.com covers every tenant, and cert-manager renews one certificate instead of 200. The Ingress gets one rule per tenant host, or one rule with a wildcard host if the app routes tenants by host header itself. I prefer one rule per tenant in the Ingress, because the rule list is the audit trail of who exists. The controller runs as 3 replicas behind the cloud load balancer. At 300 requests per second total, that is 100 per replica, well under the 500 per pod queue point from the simulator. The canary is a second Ingress with the same host pattern, the canary annotations set, and a weight of 10. The heavy tenant gets a rate limit annotation on its own Ingress rule, 20 requests per second, so its burst stops at the controller and never reaches the shared pods.
A wildcard certificate buys simplicity and costs blast radius: if it expires, all 200 tenants go dark at once, so the expiry alert pages at 30, 14, and 7 days. One shared controller buys simplicity and costs isolation: a bad annotation in one tenant's Ingress can slow the controller for all of them, so annotation changes get reviewed like code. Separate controllers per tenant would buy isolation and cost 200 controllers, which nobody wants to run. I rate limit per tenant at the Ingress, because a rejected request never reaches a pod, but I also keep a per-tenant limit inside the app, because the Ingress limit counts requests and the app limit counts work, and those are different things. And I say the canary out loud: 10 percent of traffic on the new checkout is a test, not a release, and the weight only goes up when the canary error rate stays flat for an hour.