One service calling another sounds simple. Then you have forty services, and every call needs encryption, identity, retries, and a rule about who may call whom. That work used to live in every app, written forty different ways. A service mesh moves it into a proxy that rides beside each pod. This lesson walks one call through the mesh, hop by hop, the way I trace it when two services stop talking at 2 AM.
Difficulty: Advanced · Know this first: Lesson 45 (Gateway API), Lesson 25 (Seeing Inside a Running System)
Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: in most slow requests I have chased, one or two hops eat almost the whole budget. Step through the lesson or switch the scenario and watch the bar change.
The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider or flip the mTLS switch and watch the sidecar hops, the app hop, and the total change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: the mesh tax is small per call, and it multiplies with every downstream call.
How many services run sidecars. More services mean more endpoints and more policy rules for each sidecar to sort through. The outbound sidecar hop costs 2 ms plus 1 ms per 50 services. The policy hop costs 2 ms plus 1 ms per 100 services.
How many services this one request fans out to. Each extra call pays the full mesh tax again plus the app time. The app hop is 35 ms times the number of calls.
Mutual TLS on or off. On, the handshake costs 8 ms and every hop between pods is encrypted. Off, the handshake is plain TCP at 1 ms and nothing between pods is encrypted.
Try this: (1) Leave the defaults: 50 services, 2 calls, mTLS on. The mesh tax is 17 ms per call, the total is about 124 ms, and 100 percent of the traffic is encrypted. (2) Turn mTLS off and push fan-out to 8 with 200 services. The total is about 420 ms and 0 percent is encrypted. (3) Turn mTLS back on, set services to 5 and calls to 1. The total is about 71 ms. Same mesh, same code. Only the inputs changed.
Extra diagram. I read the picture first: the boxes are the system, the moving dot is a call, and each step adds one piece until the full picture is there. No text needed to follow it.
Extra diagram. I read the picture first: the boxes are the system, the moving dot is a call with an expired sidecar certificate, and each step adds one piece until the full picture is there. No text needed to follow it.
Three questions. No score is saved anywhere — this is just you against the material.
The lesson above walks one call through the mesh, hop by hop. This section covers the same call four times, from the first proxy to the level where the mesh runs a whole fleet without eating the budget.
Definition: One service calling another sounds simple. Then you have forty services, and every call needs encryption, identity, retries, and a rule about who may call whom. That work used to live in every app, written forty different ways. A service mesh moves it into a proxy that rides beside each pod.
Types:
Uses: I use this level when two services stop talking at 2 AM. I check the proxy first, not the app. The app logs say nothing useful about a handshake that failed. The proxy logs say exactly what happened.
Dimensions:
Definition: mTLS is the proxies proving identity to each other. The retry is the proxy trying again when a call fails. The traffic rule is the split: 90 percent to the old version, 10 percent to the canary, with no deploy. Those four blocks are the mesh doing the jobs that used to be spread across forty codebases.
Types:
Uses: I use this level when a deploy needs a canary, or when a downstream service is slow and the retries are making it slower. I set the retry budget, I watch the error rate on the canary, and I change one number to roll back.
Dimensions:
Definition: Sooner or later a meshed service has to call something outside the mesh: a payment provider, a managed database, a third-party API. I do not let pods reach the internet directly. I route that traffic through an egress gateway, which is the one place that decides what may leave, terminates or originates TLS, and logs every outside call.
Types:
Uses: I use this level when the mesh has to leave the cluster, or when the sidecar bill gets too big. One door out is easier to audit than forty, and one bad node proxy touching every pod on that node is a trade I make on purpose, not by accident.
Dimensions:
Definition: At this level the mesh is a platform with a budget. The proxy resources have a limit. The control plane has an SLO. The teams know what the mesh costs, and the platform team knows which service is paying it. The mesh is invisible when it works, and it works because someone is watching it.
Types:
Uses: I use this level when the mesh bill shows up on the finance report. I ask which services pay the most and whether they use the mesh features. A proxy on a service that makes one call a minute is a tax with no benefit.
Dimensions:
Sidecar or ambient is the first trade-off. A sidecar is one proxy per pod. It is simple to reason about and it costs memory and CPU on every pod. Ambient mode moves the proxy out of the pod and onto the node, with one shared proxy per node and a second layer for policy. It is cheaper per pod and the blast radius is bigger: one bad node proxy touches every pod on that node. I pick sidecars when I want the failure to stay small, and ambient when the pod count makes the sidecar bill hurt.
Egress is the second trade-off. Sooner or later a meshed service has to call something outside the mesh: a payment provider, a managed database, a third-party API. I do not let pods reach the internet directly. I route that traffic through an egress gateway, which is the one place that decides what may leave, terminates or originates TLS, and logs every outside call. One door out is easier to audit than forty.
Federation is the third trade-off. Two clusters, one mesh, or two meshes that trust each other. The hard part is not the networking. It is trust: each cluster has its own certificate root, and each root has to trust the other. I keep the trust domains separate and write the policy so a service in one cluster cannot impersonate a service in the other. Federation done wrong is a back door with a certificate.
When not to adopt a mesh is the question I ask first. Fewer than ten services, one team, and no real traffic rules to enforce: a mesh is a platform team you do not have, solving a problem you do not have yet. A single latency-sensitive hop also argues against it, because the sidecar tax is paid on every call and there is no fan-out to hide it in. I adopt a mesh when the service count, the policy needs, or the compliance requirement make the tax worth paying.
At 10x the services, the shape holds and the control plane becomes the bottleneck. Every sidecar needs config and certificates, and the control plane pushes both. I watch push latency and the share of sidecars on the latest config. A sidecar three versions behind is a service running on rules nobody wrote anymore.
At 100x, the design changes. The mesh stops being one mesh and becomes federated meshes, one per cluster or per region, with explicit trust between them. The policy moves from per-service rules to team-owned defaults with platform guardrails. And the mesh tax gets a budget line of its own, because at that scale the tax is a real slice of p99.
# payments namespace: strict mTLS, only checkout may call fraud,
# and 10 percent of traffic goes to the canary version of pricing
# (the subsets stable and canary are defined by the DestinationRule below)
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
name: payments-strict
namespace: payments
spec:
mtls:
mode: STRICT
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: fraud-only-checkout
namespace: payments
spec:
selector:
matchLabels:
app: fraud
rules:
- from:
- source:
principals: ["cluster.local/ns/payments/sa/checkout"]
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
name: pricing
namespace: payments
spec:
host: pricing
subsets:
- name: stable
labels:
version: stable
- name: canary
labels:
version: canary
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
name: pricing-split
namespace: payments
spec:
hosts: ["pricing"]
http:
- route:
- destination:
host: pricing
subset: stable
weight: 90
- destination:
host: pricing
subset: canary
weight: 10
What the snippet means: the first block says every call inside the payments namespace must use mTLS, no exceptions. The second says only the checkout service identity may call fraud, and everyone else gets a rejection from the sidecar, not from the app. The third block is the DestinationRule for host pricing: it defines the stable subset (version: stable) and the canary subset (version: canary), so the VirtualService has something real to route to — without it, Istio cannot resolve either subset. The fourth sends 90 percent of pricing traffic to the stable version and 10 percent to the canary, with no deploy. Those four blocks are the mesh doing the jobs that used to be spread across forty codebases.
A control plane upgrade shipped a new root certificate for the mesh. Most sidecars picked it up. A set of long-lived sidecars did not. They kept trusting the old root, and every call between a new sidecar and an old one failed the handshake. The apps were healthy. The pods were healthy. The failures showed up as 503s between services that had never had a problem before, and the app logs said nothing useful about any of it.
Four things. First, control plane rollouts are canaried now, one cluster at a time, with a pause to watch handshake errors before the next. Second, an alert on handshake failure rate per service pair, so a cert problem pages before users feel it. Third, a dashboard for sidecar version and certificate age, so a stale sidecar is visible instead of invisible. Fourth, a synthetic mesh call every minute that exercises a real mTLS handshake between two services. The app dashboard can be green while the mesh is broken. The synthetic call is the only check that sees what the services see.
Design the in-mesh traffic for a checkout platform with 40 services. Checkout calls pricing, fraud, ledger, and notifications. Fraud must never be callable from the public API. Deploys must canary at 10 percent. mTLS is strict everywhere. One external payment provider sits outside the mesh.
Start with the budget from this lesson. The mesh tax is about 17 ms per call: outbound sidecar 3, handshake 8, policy 2, inbound sidecar 4. Four downstream calls pay it four times, about 68 ms. The app time is about 35 ms per service, so four calls in sequence add about 140 ms. Fixed hops add about 20 ms. That is roughly 228 ms end to end, which leaves about 70 ms of headroom under a 300 ms budget. I run pricing and fraud in parallel, not in sequence, and I move ledger and notifications after the response, so the user never waits for them. The egress call to the payment provider goes through the egress gateway with its own timeout, so a slow provider cannot hold the checkout call open.
Strict mTLS everywhere buys verified identity and costs a migration: every service has to be meshed before the strict mode turns on, so I run permissive first and watch the logs for unmeshed callers. Sidecars buy small blast radius and cost a proxy per pod; ambient buys lower cost and puts more pods behind one node proxy. Canary by weight is simple and blind; canary by header lets me canary for one user or one region, and costs a header rule. The egress gateway buys one audited door out and costs a single place that can fail, so it gets its own replicas and its own health checks.