Platform Engineering · Lesson 91

Istio Mastery

Lesson 51 gave you the mesh. This lesson gives you the tool that runs most meshes in production. Istio is a control plane and a set of rules that tell a fleet of Envoy sidecars what to do with every request. One request comes in through a gateway. A routing rule decides where it goes. A destination rule decides which version answers and how hard it may be pushed. An identity rule decides whether the caller is even allowed to ask. I learned Istio the way I learn most tools in this job. I broke a canary with a retry policy, watched the p99 climb while every dashboard stayed green, and then read the config until the config confessed. This lesson walks one checkout request through Istio, rule by rule, from a first install to the deep end: filter chains, Wasm plugins, ambient mesh, and how to debug a mesh that has stopped telling the truth.

Difficulty: Advanced · Know this first: Lesson 51 (Service Mesh)
Run it as:

When it breaks, I check

One thing to remember

The mesh, drawn as boxes

I start with one app, one sidecar, and one rule. Then I add the control plane, the certificates, the other two rules, the telemetry, and a second cluster. The picture grows one box at a time.

istiod3 rule objectsstrict mTLS2 clusters
checkout1 pod
VirtualService1 route
istiodcontrol plane
gatewayEnvoy
checkout + pricing2 sidecars
VS, DR, AuthZ3 rules
cert storerotates
cluster 2east-west

When it breaks, I check:

control/configRequest · dataResponse · the way back

The bad canary: a retry storm, hop by hop

First I see the normal road, one clean call to the stable version. Then the same request draws the canary, the canary answers slowly, and the retries pile on. The last step is the brake that stops the storm.

10% canary180 msretry budget
v1 stable40 ms
VirtualService10%
sidecarv2
v2180 ms
retry 12 calls
retriesgrowing
breakerbudget
answerlate

When it breaks, I check:

Response · the way backfailure/retry

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one checkout request, not a measurement from a live system, but the shape is honest: in most slow requests I have chased, one or two hops eat almost the whole budget. Step through the lesson or switch the scenario and watch the bar change. The broken canary triples the trip without the app doing anything different in kind. Same code path, same rules. Only the canary's speed and the retries changed, and the bill tripled.

Turn the knobs: canary, retries, and fan-out

The budget bar above shows three fixed stories. This simulator lets you write your own. It models one checkout request that fans out to pricing and the services behind it, with a canary in the mix and retries allowed — but a retry only costs time when one actually fires. The walkthrough above follows one call (76 ms), and this simulator starts from the same place: one downstream call, no canary, so its no-failure total is 76 ms — 36 ms of fixed Istio hops plus 40 ms of app work. Move a slider and watch the app hop, the retry cost, and the total change in the budget bar, in the hop timings, and in the status line. The numbers are a teaching example, not live measurements. The shape is the lesson: the mesh tax is small per call, it multiplies with every downstream call, and retries multiply it again.

How many services this one request calls behind it. The app hop is the average app time per call times this number. Every extra call pays the mesh tax again.

Share of traffic the routing rule sends to the canary version. Stable answers in 40 ms, the canary in 70 ms, so the average app time per call is 40 ms plus 0.3 ms per percent here.

How many times a call gets tried again. Each retry that actually fires costs 12 ms per downstream call, and every retry lands on the same service that was already slow. Allowed retries cost nothing until one fires — use the switch below.

Retries are allowed, but an allowed retry costs nothing until one fires. Off is the happy path: no failure, no retry, 0 ms of retry cost. On prices a request where a call failed and was retried.

A delay the mesh adds on purpose, for testing. It lands straight on the app hop. This is the knob people forget they left on.

Total trip76 msfixed hops 36 ms + app hop + retry cost
App hop40 ms40 ms average per call × 1 call
Retry cost0 msNo retry fired — retries allowed but none fired
Mesh tax per call36 ms36 ms fixed (27 ms out + 9 ms back) · no retry fired · canary share of app time ~0%

Try this: (1) Leave the defaults: 1 call, canary 0 percent, 1 retry allowed but none fired, no delay. The total is about 76 ms: 36 ms of fixed Istio hops, 40 ms of app work, and 0 ms of retry cost. (2) Push the canary to 100 percent and the retries to 3 with 8 calls, and turn the retry-fired switch on. The total is about 884 ms. Same services, same code. Only the knobs changed. (3) Set everything back, then set the fault injection delay to 200 ms and watch the app hop jump by exactly 200. That is what a forgotten fault-injection rule costs in production.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

Istio looks like one big tool. It is not. It is four layers stacked on each other, and each layer only makes sense after the one below it. I learned it in this order, and I teach it in this order: get a sidecar running and route one request, then shape the traffic, then lock it down and watch it, then open the engine itself. Skip a layer and the next one reads like magic. Build them in order and nothing in this lesson is magic.

Basic — A sidecar beside the app, and one route

Istio at the basic level is a sidecar proxy injected next to each app container, plus a control plane that tells the sidecars what to do. The app makes a plain call by service name. The sidecar intercepts it, and the control plane hands the sidecar its config: which service that name means, and which pods are healthy. Nothing in the app changes.

Types:

  • Sidecar injection, automatic by namespace label or manual per pod.
  • The ingress gateway, the Envoy proxy at the edge that takes outside traffic in.
  • A VirtualService with one route, one host, one destination, no split.
  • The control plane, istiod, which pushes config and certificates to the sidecars.

Uses: when a team has a few services and wants traffic to stop being hand-wired in app code. One route from checkout to pricing. One gateway for outside traffic. The first win is that a deploy stops breaking the address of a service, because the app calls a name and the mesh resolves it.

Dimensions: Scale: holds up to a few dozen services without special care. Failure modes: a pod without a sidecar is outside the mesh and no rule touches it, so a missing injection label is the most common basic failure; a gateway with the wrong host drops outside traffic at the door. Cost: one sidecar per pod, a small slice of CPU and memory per pod, paid whether it needs the mesh or not. Trade-offs: you pay the sidecar tax on every call before you get any advanced features, so a service that never needs routing, policy, or telemetry is paying for a mesh it does not use.

Intermediate — Shaping traffic: subsets, retries, mirroring, faults

Traffic shaping with VirtualService and DestinationRule. A VirtualService splits traffic by weight, rewrites paths, and sets timeouts and retries. A DestinationRule defines subsets, named groups of pods, usually one per version, and sets the load balancing, the connection pool, and the circuit breaker for each subset. Together they decide which version answers, how much traffic it gets, and how hard it may be pushed.

Types:

  • Traffic split by weight, for canaries.
  • Subsets by version label, stable and canary.
  • Retries with a timeout and a limit.
  • Timeouts per route.
  • Traffic mirroring, which copies a share of live requests to a second version and throws the answer away.
  • Fault injection, which adds delay or errors to a share of requests on purpose.

Uses: safe deploys and honest tests. A canary gets 10 percent of real traffic before it gets all of it. Mirroring lets a new version see real requests without any user seeing its answers. Fault injection proves a timeout works before a real failure proves it does not. Deploys stop being an event and start being a dial.

Dimensions: Scale: holds to a few hundred services, but the rule count grows fast, because every route and every subset is a rule the control plane has to push. Failure modes: a retry policy with no timeout turns one slow call into several slow calls; a subset with a stale label matches no pods and traffic falls back silently; a rewrite rule that matches too broadly eats requests meant for another route. Cost: the cost is config, not compute; each rule is cheap to run, and the bill is the time it takes a human to read all of them when something breaks. Trade-offs: every knob can hurt as easily as it helps, so change one knob at a time and watch the trace. A canary and a retry policy changed together is two variables, and two variables cannot be debugged.

Advanced — Locking it down and watching it: mTLS, policy, telemetry, canary, multi-cluster

Identity, permission, and proof. Strict mTLS gives every service a certificate identity. AuthorizationPolicy decides which identity may call which endpoint. Telemetry from every sidecar records each call, its duration, and its result. Canary releases with Flagger automate the weight shift: Flagger watches the metrics and moves the weight up, or rolls it back, without a human in the loop. Multi-cluster extends the same mesh across clusters, with each cluster trusting the other's certificate root.

Types:

  • PeerAuthentication, strict or permissive mTLS per namespace or per workload.
  • AuthorizationPolicy, allow or deny by service identity, namespace, or method.
  • Telemetry: metrics, traces, and access logs from the sidecars.
  • Flagger canary releases, automated weight shifts with metric gates.
  • Multi-cluster meshes, one control plane or one per cluster, with a shared trust bundle.
  • Egress gateways, the one door out to services outside the mesh.

Uses: when the mesh has to prove something, not just move traffic. A payments service that must only answer checkout. A canary that must roll back on its own when the error rate climbs. Two clusters that must call each other without opening the whole network. The mesh becomes the enforcement point, and the app stops being asked to enforce anything.

Dimensions: Scale: the control plane becomes the bottleneck; watch push latency and the share of sidecars on the latest config. Failure modes: a policy in permissive mode logs a violation and lets the call through, so a team can believe it is protected while it is not; a Flagger gate on the wrong metric promotes a bad canary because the metric never moved; a multi-cluster trust bundle missing one root fails handshakes exactly like an expired certificate. Cost: operational; certificates rotate daily, policies need owners, and the telemetry has to go somewhere and be read by someone. Trade-offs: strict mTLS buys verified identity and costs a migration, because every service has to be meshed before strict mode turns on. Automation buys faster canaries and costs a gate you have to trust. Multi-cluster buys one mesh across regions and costs two roots of trust to keep straight.

The AuthorizationPolicy that denied everyone. An AuthorizationPolicy that denies everyone is a policy that works too well. I test it from a service that should be allowed and from one that should not, because a policy that blocks the good traffic is a policy that breaks the app.

Do this: apply the policy from the snippet, then call the endpoint from an allowed service and from a denied one. Expected: the allowed call succeeds, and the denied call gets a 403. Remove the policy, and expect both to succeed. The 403 is the policy working, and the test from both sides is the proof.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: shop-api-policy
  namespace: shop
spec:
  selector:
    matchLabels:
      app: shop-api
  rules:
    - from:
        - source:
            principals:
              - cluster.local/ns/shop/sa/frontend

What the snippet means: the policy allows only the frontend service account to call the API. A call from any other identity gets a 403, and the policy is the boundary between the services.

The canary that Flagger rolled back on its own. Flagger watches the metrics and moves the weight up, or rolls it back, without a human in the loop. The canary is a test with real traffic, and the metric gate is the judge. I watch the first canary closely, because a gate that is too loose promotes a bad version, and a gate that is too tight never promotes anything.

Do this: deploy a canary with a bad version, and watch Flagger. Expected: the weight rises a little, the error rate rises, and Flagger rolls the weight back to zero on its own. Deploy a good version, and expect the weight to rise to full. The rollback is the automation working, and the metric is the judge.

kubectl get canary -n shop -w
# expect the weight to rise, then fall back to zero when the error rate trips the gate

What the snippet means: the canary status shows the weight and the phase. A canary that rolls back on its own is a canary that works, and the metric gate is the line it must not cross.

Read the flags: kubectl get canary -n -w

A Canary is a Flagger object that moves traffic weight. Get with -w watches it, so I see the weight rise and then fall back to zero when the error rate trips the gate.

  • kubectl get canary: This lists the Canary objects. Each line is one canary rollout. Without a watch flag, get prints once and exits.
  • -n shop: The -n flag picks the namespace. The canary lives in shop, not in the current namespace. Without it, kubectl looks in the wrong place and finds nothing.
  • -w: This is the short form of --watch. It keeps the command running and prints each change as it happens. Without it, I would have to run get again and again to see the weight move.
  • --watch-only (added): The lesson did not show this one yet. It watches without printing the first list. I use it when the first list is long and I only care about what changes next.

Try it: run kubectl get canary -n shop -w. Expect the weight to rise, then fall back to zero when the error rate trips the gate. Stop the watch with Ctrl-C when the weight settles.

Master — Inside the engine: filter chains, Wasm, ambient, debugging

The Envoy engine underneath Istio. A filter chain is the ordered list of filters a request passes through inside Envoy: the listener takes the connection, the filters run in order, and each filter can read, change, or reject the request. A Wasm plugin is custom code, usually written in Rust or Go, compiled to WebAssembly, that runs as a filter inside Envoy, so you can add behavior Istio does not ship. Ambient mesh moves the proxy out of the pod and onto the node: a shared ztunnel handles identity and encryption per node, and a waypoint proxy handles policy per service. Debugging a broken mesh is the skill of reading what the data plane actually has, not what the config says it should have.

Types:

  • Envoy filter chains: network filters, HTTP filters, and the order they run in.
  • EnvoyFilter resources, which patch the generated Envoy config.
  • Wasm plugins, custom filters loaded into Envoy.
  • Ambient mesh: ztunnel per node, waypoint proxy per service or namespace.
  • Debugging tools: istioctl proxy-config, istioctl analyze, istioctl authn tls-check, and the Envoy admin interface.
  • Telemetry pipelines: the sidecar metrics and traces, and the backends that store them.

Uses: when Istio's shipped behavior is not enough. A custom header check no stock filter does. A per-request rule that lives in code, not YAML. An ambient mesh when the pod count makes the sidecar bill hurt. And the debugging tools every time the mesh stops telling the truth: when the config says one thing and the traffic does another, the proxy config is the truth and the YAML is the wish.

Dimensions: Scale: holds at thousands of services, because the cost moves from per-pod proxies to per-node proxies in ambient mode, and the config push moves from every sidecar to fewer, bigger proxies. Failure modes: a Wasm plugin that crashes takes the filter chain with it, so a bad plugin is a data-plane outage, not a config error; an EnvoyFilter that patches the wrong filter chain silently does nothing, or silently breaks the chain it did patch; an ambient waypoint that goes down takes policy for a whole service with it, because the policy no longer lives in the pod. Cost: expertise; Wasm plugins need a build pipeline, a test harness, and an owner; ambient mode needs a team that understands two proxies instead of one. Trade-offs: custom filters buy behavior Istio does not ship and cost a second codebase to maintain, test, and roll back. Ambient buys lower per-pod cost and costs a bigger blast radius per proxy. Debugging at this layer buys the truth and costs time, because the truth lives in generated config, and generated config is not written to be read by humans.

EnvoyFilter: the patch that changed the generated config. An EnvoyFilter patches the config Istio generates, and a patch that is wrong breaks the proxy in a way that looks like a networking problem. I read the generated config before and after the patch, because the diff is the truth and the patch is the guess.

Do this: apply an EnvoyFilter, then dump the proxy config with the command in the snippet, and find the change. Expected: the patched value appears in the dump, and nothing else changed. Remove the filter, dump again, and expect the original config back. The dump is the proof, and the patch is the change.

istioctl proxy-config listener my-pod -o json | python3 -c "import json,sys; d=json.load(sys.stdin); print([l['name'] for l in d])"

What the snippet means: the dump is the config the proxy actually runs. A patch that appears in the dump is a patch that took effect, and a patch that does not appear is a patch that did nothing.

Read the flags: istioctl proxy-config listener, cluster, and route

The YAML is the wish. The proxy config is the truth. Proxy-config reads the live Envoy config from the pod, so I can see what the proxy actually does, not what the YAML asked for.

  • istioctl proxy-config listener my-pod: This prints the listeners on the pod proxy. A listener is a port the proxy accepts traffic on. The list names each one, so I can see whether the port I expect is really listening.
  • -o json: The -o flag sets the output format. Json prints the config as JSON, which a script can read. Without it, proxy-config prints a table, and the table hides the detail.
  • | python3 -c: The pipe hands the JSON to Python. The -c flag runs the code string. The code prints the listener names, so I can read them in one look.
  • istioctl proxy-config cluster my-pod | grep shop-api: Cluster prints the upstream clusters the proxy can call. The pipe to grep keeps only the shop-api cluster. A missing cluster means the proxy has no route to the service.
  • istioctl proxy-config route my-pod | grep shop-api: Route prints the routing rules. The pipe to grep keeps only the shop-api routes. A missing route means the proxy has nowhere to send the request.
  • istioctl proxy-config endpoint (added): The lesson did not show this one yet. It prints the endpoints behind a cluster, like istioctl proxy-config endpoint my-pod. I use it when the cluster exists but the endpoints are empty, which means no healthy pods behind it.

Try it: run istioctl proxy-config listener my-pod -o json and pipe it to Python to print the listener names. Run the cluster and route commands next and grep for shop-api. Expect the cluster and the route to exist.

Debugging the mesh: read what the data plane has. Debugging a broken mesh is the skill of reading what the data plane actually has, not what the config says it should have. I dump the proxy config, I read the clusters and the routes, and I compare them with the config I wrote, because the difference between the two is the bug.

Do this: when a route does not work, dump the proxy clusters with the command in the snippet, and find the service. Expected: the cluster list shows the service, or it does not. If it does not, the config never reached the proxy, and the control plane is the problem. If it does, the route is the problem, and the route table is the next place to look.

istioctl proxy-config cluster my-pod | grep shop-api
istioctl proxy-config route my-pod | grep shop-api

What the snippet means: the two dumps are the clusters and the routes the proxy actually has. A service missing from the clusters never got its config, and a route missing from the routes never got its rule.

Go deeper

The steps above are one request through nine hops. Production is the rest of it. Here is what I actually spend my time on once the rules are written and the canary is running.

Filter chain order

The filter chain is the order of operations inside Envoy, and the order is the whole lesson. A request enters the listener, then runs through the HTTP filters in the order they are listed. The routing filter is usually near the end, and the auth filter runs before it. If you add a custom filter after the routing filter, it never sees the request, because the request is already gone. I have debugged a filter that did nothing for an hour before I read the order and found it was listed after the filter that ends the chain. The chain is not a list of features. It is a pipeline, and a pipeline runs in order.

Wasm plugins

A Wasm plugin is custom code that runs inside Envoy as a filter. You write it in Rust or Go, compile it to WebAssembly, and load it into the proxy. I use it for exactly one kind of job: a check no stock filter does, like a custom header rule or a per-request transform the business needs. The warning is the price. A plugin is a second codebase. It needs a build pipeline, a test harness, and an owner, and a plugin that crashes takes the filter chain with it. I do not write a Wasm plugin for something a VirtualService or an AuthorizationPolicy already does. I write one when the shipped filters cannot do the job, and I keep it small, because a small plugin is a small blast radius.

Ambient mesh

Ambient mesh moves the proxy out of the pod. A ztunnel runs on each node and handles identity and encryption for every pod on that node. A waypoint proxy handles policy, one per service or per namespace. The win is cost: no sidecar per pod, so a thousand pods stop paying a thousand proxies. The trade is blast radius: one bad node proxy touches every pod on that node, and one bad waypoint touches a whole service. I pick ambient when the pod count makes the sidecar bill hurt and the team is ready to debug two proxies instead of one. I stay on sidecars when I want the failure to stay small. There is no free proxy. There is only a choice about where the proxy lives and how many pods it can hurt.

Debugging a broken mesh

When the mesh breaks, I stop reading the YAML and start reading the data plane. The YAML is the wish. The proxy config is the truth. I run istioctl proxy-config cluster and istioctl proxy-config route on the sidecar that is misbehaving, and I compare what Envoy actually has with what the VirtualService says it should have. If the config is stale, the control plane never pushed it, and the bug is in the push, not in the rule. I run istioctl analyze on the namespace, because it catches a VirtualService that references a subset that does not exist, and that is a failure that never shows up in the app logs. And I run istioctl authn tls-check between two services, because it shows the mTLS mode each side actually has, and a pair that disagrees fails the handshake while both sides look fine on their own.

What changes at 10x and 100x

At 10x the services, the control plane becomes the bottleneck. Every sidecar needs config and certificates, and the control plane pushes both. I watch push latency and the share of sidecars on the latest config. A sidecar three versions behind is a service running on rules nobody wrote anymore. At 100x, the design changes. The mesh stops being one mesh and becomes federated meshes, one per cluster or per region, with explicit trust between them. The policy moves from per-service rules to team-owned defaults with platform guardrails. And the mesh tax gets a budget line of its own, because at that scale the tax is a real slice of p99, and a tax nobody budgets for is a tax nobody controls.

The three objects behind this lesson

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: pricing
spec:
  hosts: [pricing]
  http:
  - route:
    - destination: { host: pricing, subset: stable }
      weight: 90
    - destination: { host: pricing, subset: canary }
      weight: 10
    retries:
      attempts: 1
      perTryTimeout: 2s
    timeout: 5s
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: pricing
spec:
  host: pricing
  trafficPolicy:
    loadBalancer:
      simple: LEAST_CONN
    connectionPool:
      tcp: { maxConnections: 100 }
      http: { http1MaxPendingRequests: 50 }
    outlierDetection:
      consecutive5xxErrors: 5
      interval: 10s
      baseEjectionTime: 30s
  subsets:
  - name: stable
    labels: { version: stable }
  - name: canary
    labels: { version: canary }
---
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: checkout
spec:
  mtls:
    mode: STRICT

Three objects, three jobs, one request. The VirtualService sends 90 percent of pricing traffic to stable and 10 percent to canary, allows one retry with a 2 second per-try timeout, and cuts the whole call off at 5 seconds. The DestinationRule says what stable and canary are, by version label, balances by least connections, caps the pool at 100 connections and 50 pending requests, and ejects a pod after 5 straight 5xx errors for 30 seconds. That ejection rule is the circuit breaker, and it is the thing that stops a retry policy from hammering a sick canary. The PeerAuthentication says every call in the checkout namespace must use mTLS, no exceptions. Read them together and you can see the whole trip: route, pick, protect. Change one of them without reading the other two and you have changed the trip in a way you did not intend.

Wasm plugins: custom code in the filter chain

A Wasm plugin is custom code that runs as a filter inside Envoy, so you can add behaviour Istio does not ship. It is powerful, and it is a risk: a plugin that crashes takes the proxy with it, and a plugin that is slow makes every request slow. I test the plugin in a canary first, with the metrics open.

Do this: deploy a Wasm plugin to one workload, and watch the request latency. Expected: the latency stays flat, and the plugin does its job. Deploy a plugin with a bug, and expect the latency to rise or the requests to fail. The canary is the test, and the metric is the judge. A plugin that is not measured is a plugin that is trusted, and trust is not a test.

kubectl get wasmplugin -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name
# deploy to one workload first, watch latency, then roll out

What the snippet means: the list shows every Wasm plugin and where it runs. A plugin on one workload is a test. A plugin on every workload is a commitment, and the commitment comes after the test.

Read the flags: kubectl get wasmplugin -A -o custom-columns

A WasmPlugin is a filter that runs inside Envoy. Get lists them across all namespaces, and custom columns print the namespace and name in one table.

  • kubectl get wasmplugin: This lists the WasmPlugin objects. Each line is one plugin. Without a format flag, the table hides the details.
  • -A: This is the short form of --all-namespaces. It lists objects across all namespaces. Without it, get shows only the current namespace.
  • -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name: The -o flag sets the output format. Custom-columns prints chosen fields as columns. The part before each colon is the column header. The part after is the JSON path to the field.
  • --sort-by (added): The lesson did not show this one yet. It sorts the list by a field, like --sort-by=.metadata.name. I use it when the list is long and I want it in order.

Try it: run kubectl get wasmplugin -A -o custom-columns. Expect one row per plugin. Deploy a plugin to one workload first, watch latency, then roll it out wider.

One time it broke

This one stayed with me because every dashboard stayed green while the users waited. The canary was slow, the retries were loyal, and the mesh did exactly what it was told. This is an illustrative scenario built from a real SRE pattern, anonymized, with no company names and no real system names.

What broke

A checkout service called pricing through Istio. A canary version of pricing shipped with an extra lookup nobody noticed in review. The canary answered in about 180 ms instead of 40. The VirtualService gave the canary 10 percent of traffic, and the retry policy allowed 3 retries with no per-try timeout. A request that drew the canary waited for the slow answer, retried it, waited again, and retried again. One slow call became four slow calls, all landing on the same struggling canary pods. The circuit breaker never opened, because slow is not an error, and the outlier detection only counts 5xx responses. Checkout p99 climbed past 300 ms on one request in ten. The app dashboards stayed green, because pricing was answering, just slowly, and the mesh dashboards stayed green, because every retry eventually got an answer.

Timeline

  • 10:02Canary deploy of pricing goes out at 10 percent weight, extra lookup ships with it.
  • 10:14Checkout p99 starts climbing on one request in ten; the other nine are fine, so the alert does not fire.
  • 10:31First user report, checkout feels slow sometimes; on-call checks the app dashboards, all green.
  • 10:47On-call reads the sidecar trace for a slow checkout: one pricing call, three retries, all to the canary subset, all slow.
  • 11:05Extra lookup found in the canary code; retry policy found in the VirtualService: 3 attempts, no per-try timeout.
  • 11:12Canary weight set to zero; checkout p99 falls back under 80 ms in two minutes.
  • 11:40Retry policy changed to 1 attempt with a 2 second per-try timeout, and outlier detection added on latency, not just on 5xx errors.
  • 12:00Canary reintroduced at 5 percent with the new policy; it stays slow, the breaker opens, and the weight rolls back on its own.

What changed after

Three things. First, every retry policy now has a per-try timeout and a limit of one, unless a human writes down why it needs more. A retry without a timeout is a way to wait longer for the same failure. Second, canary gates watch latency, not just errors. A canary that answers slowly is a bad canary, even if every answer is a 200. Third, Flagger runs the canary now, with the weight shift automated and the rollback automatic when the latency gate trips. A human still approves the deploy. No human has to watch the p99 at 2 AM to notice the canary is slow. The mesh did what it was told in this incident. The lesson is that what it was told was wrong, and the config was the bug.

If you remember nothing else

Design exercise

This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real mesh. Take a minute with it before you read my approach.

The prompt

Design the Istio setup for a checkout platform with 60 services across 2 clusters. Checkout calls pricing, fraud, ledger, and notifications. Fraud must never be callable from the public API. Deploys must canary at 10 percent and roll back on their own when latency climbs. mTLS is strict everywhere. One external payment provider sits outside the mesh. The latency budget is 300 ms at p99 for a checkout request. Sketch the gateway, the routing rules, the destination rules, the policies, and the multi-cluster trust, and say what you would measure.

Clarifying questions to ask first

  • Which calls are on the critical path for the checkout response, and which can happen after the answer?
  • How fresh must the fraud score be, and what happens when the fraud service is down: block checkout or let it through?
  • Is the 300 ms budget for one cluster or for a call that crosses to the second cluster, and how much of it can the multi-cluster road take?
  • Who owns the policies: one platform team, or does each service team write its own, and who reviews a policy change?
  • What happens when the external payment provider is slow: does checkout wait, or queue and confirm later, and where does that queue live?

Rough sizing, worked with Little's Law

Start with the budget from this lesson. One checkout request takes about 76 ms inside one cluster: fixed Istio hops about 36 ms — 27 ms on the way out and 9 ms on the way back — and the app about 40 ms. Four downstream calls in sequence would be about 304 ms, which is already over the 300 ms budget, so I run pricing and fraud in parallel, not in sequence, and I move ledger and notifications after the response, so the user never waits for them. The parallel pair costs about 76 ms plus the slower of the two apps, roughly 120 ms end to end, which leaves about 180 ms of headroom. Little's Law for the sizing: at 500 checkout requests a second, each spending 0.076 seconds inside the mesh path, the number inside the system at once is 500 times 0.076, which is 38 concurrent requests. At a peak of 1,500 requests a second it is 1,500 times 0.076, which is 114 concurrent requests. I size the ingress gateway and the sidecar connection pools for 114 concurrent, plus headroom, and I alert at 80 percent of the pool. A multi-cluster call takes about 133 ms instead of 76, so the same 500 requests a second across clusters is 500 times 0.133, which is about 67 concurrent, and the gateway pool for the cross-cluster road is sized for that number, not for the single-cluster one.

Trade-offs to name

Strict mTLS everywhere buys verified identity and costs a migration: every service has to be meshed before strict mode turns on, so I run permissive first and watch the logs for unmeshed callers. Sidecars buy a small blast radius and cost a proxy per pod; ambient buys lower cost and puts more pods behind one node proxy. Canary by weight is simple and blind; Flagger buys automatic rollback and costs a metric gate I have to trust. The egress gateway buys one audited door out to the payment provider and costs a single place that can fail, so it gets its own replicas and its own health checks. And the target is p99, not the average. An average of 120 ms with a p99 of 400 ms fails this design, and the p99 is where the slow canary and the retries live.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series
Lesson 108 — Linkerd