Platform Engineering · Lesson 51

Service Mesh

One service calling another sounds simple. Then you have forty services, and every call needs encryption, identity, retries, and a rule about who may call whom. That work used to live in every app, written forty different ways. A service mesh moves it into a proxy that rides beside each pod. This lesson walks one call through the mesh, hop by hop, the way I trace it when two services stop talking at 2 AM.

Difficulty: Advanced · Know this first: Lesson 45 (Gateway API), Lesson 25 (Seeing Inside a Running System)

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: in most slow requests I have chased, one or two hops eat almost the whole budget. Step through the lesson or switch the scenario and watch the bar change.

Try it: change the inputs and watch the trip bend

The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider or flip the mTLS switch and watch the sidecar hops, the app hop, and the total change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: the mesh tax is small per call, and it multiplies with every downstream call.

50

How many services run sidecars. More services mean more endpoints and more policy rules for each sidecar to sort through. The outbound sidecar hop costs 2 ms plus 1 ms per 50 services. The policy hop costs 2 ms plus 1 ms per 100 services.

2

How many services this one request fans out to. Each extra call pays the full mesh tax again plus the app time. The app hop is 35 ms times the number of calls.

Mutual TLS on or off. On, the handshake costs 8 ms and every hop between pods is encrypted. Off, the handshake is plain TCP at 1 ms and nothing between pods is encrypted.

Try this: (1) Leave the defaults: 50 services, 2 calls, mTLS on. The mesh tax is 17 ms per call, the total is about 124 ms, and 100 percent of the traffic is encrypted. (2) Turn mTLS off and push fan-out to 8 with 200 services. The total is about 420 ms and 0 percent is encrypted. (3) Turn mTLS back on, set services to 5 and calls to 1. The total is about 71 ms. Same mesh, same code. Only the inputs changed.

The mesh, from scratch: two services, two sidecars

Extra diagram. I read the picture first: the boxes are the system, the moving dot is a call, and each step adds one piece until the full picture is there. No text needed to follow it.

Two proxies on every callPolicy decided in one placeEgress gateway is the one door out
12345678

ServicesControl planeOut of the meshMoving dots: request · response

The call that fails while both apps are healthy

Extra diagram. I read the picture first: the boxes are the system, the moving dot is a call with an expired sidecar certificate, and each step adds one piece until the full picture is there. No text needed to follow it.

Healthy apps, failed callCheck the proxy logs first at 2 AMCanary split 90 to 10, rolled back with one number
12345678

Healthy appsThe failureOut of the meshMoving dots: request · response

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

The lesson above walks one call through the mesh, hop by hop. This section covers the same call four times, from the first proxy to the level where the mesh runs a whole fleet without eating the budget.

Basic — A proxy beside every pod

Definition: One service calling another sounds simple. Then you have forty services, and every call needs encryption, identity, retries, and a rule about who may call whom. That work used to live in every app, written forty different ways. A service mesh moves it into a proxy that rides beside each pod.

Types:

  • The sidecar — one proxy per pod, doing the encryption, the retries, and the rules.
  • mTLS — the proxies prove who they are to each other, on every call.
  • The retry — the proxy retries a failed call, so the app does not have to.
  • The policy — who may call whom, decided in one place, not in forty codebases.

Uses: I use this level when two services stop talking at 2 AM. I check the proxy first, not the app. The app logs say nothing useful about a handshake that failed. The proxy logs say exactly what happened.

Dimensions:

  • Load: At two services, the proxy is overhead. At forty, the proxy is the only way the rules stay the same everywhere. The cost is paid per pod, so the bill grows with the fleet.
  • Breaks: It breaks when the proxy fails and the app is healthy. The pods are fine, the calls fail, and the app logs say nothing. I notice it in the 503s between services that never had a problem before.
  • Cost: The cost is memory and CPU on every pod, plus the control plane that configures the proxies. A sidecar per pod is simple to reason about and it costs on every pod.
  • Trade-off: I give up a little speed and a little memory to get rules that live in one place. Every call pays the proxy tax. In return, no app has to write encryption, retries, and policy again.

Intermediate — mTLS, retries, and traffic rules

Definition: mTLS is the proxies proving identity to each other. The retry is the proxy trying again when a call fails. The traffic rule is the split: 90 percent to the old version, 10 percent to the canary, with no deploy. Those four blocks are the mesh doing the jobs that used to be spread across forty codebases.

Types:

  • mTLS with rotation — certificates that renew themselves, so no call fails on an expired cert.
  • Retry with a budget — retries that stop, so a retry storm does not take down the service it is calling.
  • The canary split — a weight change, no deploy, watched before the rest follows.
  • The circuit breaker — the proxy stops calling a service that keeps failing, and lets it recover.

Uses: I use this level when a deploy needs a canary, or when a downstream service is slow and the retries are making it slower. I set the retry budget, I watch the error rate on the canary, and I change one number to roll back.

Dimensions:

  • Load: At low traffic, a retry is harmless. At high traffic, a retry multiplies the load on a service that is already struggling, and the retries become the outage.
  • Breaks: It breaks when a certificate does not rotate, or when the retry budget is too generous. I notice the first when calls between new and old proxies fail the handshake, and the second when the error rate climbs and the retries climb with it.
  • Cost: The cost is the proxy CPU on every call, and the control plane that hands out certificates. The certificates are free. The rotation that fails is expensive.
  • Trade-off: I give up the app's control over retries to get one policy everywhere. The app cannot retry forever. The mesh will not let it. That is the point.

Advanced — Egress, federation, and ambient

Definition: Sooner or later a meshed service has to call something outside the mesh: a payment provider, a managed database, a third-party API. I do not let pods reach the internet directly. I route that traffic through an egress gateway, which is the one place that decides what may leave, terminates or originates TLS, and logs every outside call.

Types:

  • The egress gateway — one door out, easier to audit than forty.
  • Federation — two clusters, one mesh, or two meshes that trust each other. The hard part is trust, not networking.
  • Ambient mode — the proxy moves out of the pod and onto the node. Cheaper per pod, bigger blast radius.
  • The sidecar choice — sidecars when I want the failure to stay small, ambient when the pod count makes the sidecar bill hurt.

Uses: I use this level when the mesh has to leave the cluster, or when the sidecar bill gets too big. One door out is easier to audit than forty, and one bad node proxy touching every pod on that node is a trade I make on purpose, not by accident.

Dimensions:

  • Load: At a few pods, the sidecar bill is small. At thousands of pods, the sidecars cost more than the apps, and ambient starts to look right. The saving is real, and the blast radius is bigger.
  • Breaks: It breaks when a control plane upgrade ships a new root certificate and a set of long-lived sidecars does not pick it up. They keep trusting the old root, and every call between a new sidecar and an old one fails the handshake. I notice it in the 503s, and the app logs say nothing.
  • Cost: The cost is the control plane, the gateways, and the proxy resources. Ambient saves per-pod cost and adds a shared proxy per node that has to be watched like a service.
  • Trade-off: I give up the small blast radius of a sidecar to get a smaller bill. One bad node proxy touches every pod on that node. I make that trade when the pod count forces it, and I watch the node proxy like a service.

Master — Mesh at scale without the bill eating you

Definition: At this level the mesh is a platform with a budget. The proxy resources have a limit. The control plane has an SLO. The teams know what the mesh costs, and the platform team knows which service is paying it. The mesh is invisible when it works, and it works because someone is watching it.

Types:

  • The proxy budget — CPU and memory per proxy, measured, with a limit that stops the bill growing.
  • The control plane SLO — config push time, certificate rotation, and the push that fails.
  • The cost report — the mesh bill per team, shown next to the compute bill.
  • The exit rule — a service that does not need the mesh does not get a proxy.

Uses: I use this level when the mesh bill shows up on the finance report. I ask which services pay the most and whether they use the mesh features. A proxy on a service that makes one call a minute is a tax with no benefit.

Dimensions:

  • Load: At this scale the control plane is the bottleneck. A config push to ten thousand proxies takes time, and a bad push touches all of them. I roll the push, and I watch the error rate while it lands.
  • Breaks: It breaks when a config push fails halfway, or when the certificate root rotates and a set of proxies is left behind. I notice it when the calls between two versions of the mesh fail, and the apps on both sides are healthy.
  • Cost: The cost is the proxies, the control plane, and the platform team. It is the most expensive part of this lesson, and it is the price of forty services that do not each write their own security.
  • Trade-off: I give up the freedom to mesh everything to get a bill I can defend. Not every service needs a proxy. The ones that do get one. The ones that do not get a simpler path.

Go deeper

Sidecar or ambient is the first trade-off. A sidecar is one proxy per pod. It is simple to reason about and it costs memory and CPU on every pod. Ambient mode moves the proxy out of the pod and onto the node, with one shared proxy per node and a second layer for policy. It is cheaper per pod and the blast radius is bigger: one bad node proxy touches every pod on that node. I pick sidecars when I want the failure to stay small, and ambient when the pod count makes the sidecar bill hurt.

Egress is the second trade-off. Sooner or later a meshed service has to call something outside the mesh: a payment provider, a managed database, a third-party API. I do not let pods reach the internet directly. I route that traffic through an egress gateway, which is the one place that decides what may leave, terminates or originates TLS, and logs every outside call. One door out is easier to audit than forty.

Federation is the third trade-off. Two clusters, one mesh, or two meshes that trust each other. The hard part is not the networking. It is trust: each cluster has its own certificate root, and each root has to trust the other. I keep the trust domains separate and write the policy so a service in one cluster cannot impersonate a service in the other. Federation done wrong is a back door with a certificate.

When not to adopt a mesh is the question I ask first. Fewer than ten services, one team, and no real traffic rules to enforce: a mesh is a platform team you do not have, solving a problem you do not have yet. A single latency-sensitive hop also argues against it, because the sidecar tax is paid on every call and there is no fan-out to hide it in. I adopt a mesh when the service count, the policy needs, or the compliance requirement make the tax worth paying.

At 10x the services, the shape holds and the control plane becomes the bottleneck. Every sidecar needs config and certificates, and the control plane pushes both. I watch push latency and the share of sidecars on the latest config. A sidecar three versions behind is a service running on rules nobody wrote anymore.

At 100x, the design changes. The mesh stops being one mesh and becomes federated meshes, one per cluster or per region, with explicit trust between them. The policy moves from per-service rules to team-owned defaults with platform guardrails. And the mesh tax gets a budget line of its own, because at that scale the tax is a real slice of p99.

# payments namespace: strict mTLS, only checkout may call fraud,
# and 10 percent of traffic goes to the canary version of pricing
# (the subsets stable and canary are defined by the DestinationRule below)
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: payments-strict
  namespace: payments
spec:
  mtls:
    mode: STRICT
---
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: fraud-only-checkout
  namespace: payments
spec:
  selector:
    matchLabels:
      app: fraud
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/payments/sa/checkout"]
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: pricing
  namespace: payments
spec:
  host: pricing
  subsets:
  - name: stable
    labels:
      version: stable
  - name: canary
    labels:
      version: canary
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: pricing-split
  namespace: payments
spec:
  hosts: ["pricing"]
  http:
  - route:
    - destination:
        host: pricing
        subset: stable
      weight: 90
    - destination:
        host: pricing
        subset: canary
      weight: 10

What the snippet means: the first block says every call inside the payments namespace must use mTLS, no exceptions. The second says only the checkout service identity may call fraud, and everyone else gets a rejection from the sidecar, not from the app. The third block is the DestinationRule for host pricing: it defines the stable subset (version: stable) and the canary subset (version: canary), so the VirtualService has something real to route to — without it, Istio cannot resolve either subset. The fourth sends 90 percent of pricing traffic to the stable version and 10 percent to the canary, with no deploy. Those four blocks are the mesh doing the jobs that used to be spread across forty codebases.

One time it broke

What broke

A control plane upgrade shipped a new root certificate for the mesh. Most sidecars picked it up. A set of long-lived sidecars did not. They kept trusting the old root, and every call between a new sidecar and an old one failed the handshake. The apps were healthy. The pods were healthy. The failures showed up as 503s between services that had never had a problem before, and the app logs said nothing useful about any of it.

Timeline

  • 14:02,the control plane upgrade started, new root certificate included.
  • 14:20,503s began between a subset of service pairs. App dashboards stayed green.
  • 14:28,on-call found handshake failures in the sidecar logs. The app logs had no trace of the calls at all.
  • 14:35,the pattern was clear: sidecars that missed the config push still trusted the old root.
  • 14:50,the control plane was rolled back and the certificates were re-pushed to the stale sidecars.
  • 15:10,handshake errors fell to zero.
  • 15:30,confirmed clean across three service pairs, and the incident was closed.

What changed after

Four things. First, control plane rollouts are canaried now, one cluster at a time, with a pause to watch handshake errors before the next. Second, an alert on handshake failure rate per service pair, so a cert problem pages before users feel it. Third, a dashboard for sidecar version and certificate age, so a stale sidecar is visible instead of invisible. Fourth, a synthetic mesh call every minute that exercises a real mTLS handshake between two services. The app dashboard can be green while the mesh is broken. The synthetic call is the only check that sees what the services see.

If you remember nothing else

Design exercise

Design the in-mesh traffic for a checkout platform with 40 services. Checkout calls pricing, fraud, ledger, and notifications. Fraud must never be callable from the public API. Deploys must canary at 10 percent. mTLS is strict everywhere. One external payment provider sits outside the mesh.

Clarifying questions to ask

  • Which calls are on the critical path for the checkout response, and which can happen after the answer?
  • How fresh must the fraud score be, and what happens when the fraud service is down: block checkout or let it through?
  • What is the latency budget, and how much of it can the mesh tax take at four downstream calls?
  • Who owns the policies: one platform team, or does each service team write its own?
  • What happens when the external payment provider is slow: does checkout wait, or queue and confirm later?

Rough sizing

Start with the budget from this lesson. The mesh tax is about 17 ms per call: outbound sidecar 3, handshake 8, policy 2, inbound sidecar 4. Four downstream calls pay it four times, about 68 ms. The app time is about 35 ms per service, so four calls in sequence add about 140 ms. Fixed hops add about 20 ms. That is roughly 228 ms end to end, which leaves about 70 ms of headroom under a 300 ms budget. I run pricing and fraud in parallel, not in sequence, and I move ledger and notifications after the response, so the user never waits for them. The egress call to the payment provider goes through the egress gateway with its own timeout, so a slow provider cannot hold the checkout call open.

Trade-offs to name

Strict mTLS everywhere buys verified identity and costs a migration: every service has to be meshed before the strict mode turns on, so I run permissive first and watch the logs for unmeshed callers. Sidecars buy small blast radius and cost a proxy per pod; ambient buys lower cost and puts more pods behind one node proxy. Canary by weight is simple and blind; canary by header lets me canary for one user or one region, and costs a header rule. The egress gateway buys one audited door out and costs a single place that can fail, so it gets its own replicas and its own health checks.

Next in this series
Lesson 91 — Istio Mastery