Lesson 51 gave you the mesh. Lesson 91 gave you Istio, the big one. This lesson gives you Linkerd, the small one, and small is the whole point. Linkerd is a service mesh built to be light: tiny proxies written in Rust, mTLS on by default with no mode to forget, and a control plane that does less, on purpose. One checkout request calls pricing, and Linkerd does to it what every mesh does: it encrypts the call, checks the policy, picks a pod, and records the trip. The difference is how little it takes to do it. I learned Linkerd the way I learn most tools in this job. I ran it next to Istio on the same services, watched the same request take the same trip, and then read the dashboards until the difference confessed. This lesson walks one checkout request through Linkerd, hop by hop, from a first install to the deep end: traffic splitting, multicluster, policy, and how to debug a mesh that is too quiet to tell you what is wrong.
Difficulty: Advanced · Know this first: Lesson 51, Lesson 91
I start with one app, one proxy, and mTLS on by default. Then I add the identity system, the destination controller, the policy, the telemetry, and a second cluster with its own control plane.
When it breaks, I check:
First the single-cluster call, in one hop. Then the call crosses the gap: two handshakes, two policy checks, one answer, and a receipt from both sides.
When it breaks, I check:
Every hop charges rent in milliseconds. The bar below is a worked example for one checkout request, not a measurement from a live system, but the shape is honest: in most slow requests I have chased, one or two hops eat almost the whole budget. Step through the lesson or switch the scenario and watch the bar change. The multicluster call nearly doubles the trip without the app doing anything different in kind. Same code path, same rules. Only the road changed, and the bill nearly doubled. The slow destination lookup adds a third more, and the app never changed at all.
The budget bar above shows three fixed stories. This simulator lets you write your own. It models one checkout request that fans out to pricing and the services behind it, with a canary in the mix and retries on, across one or more clusters. Move a slider and watch the app hop, the retry cost, and the total change in the budget bar, in the hop timings, and in the status line. The numbers are a teaching example, not live measurements. The shape is the lesson: the mesh tax is small per call, it multiplies with every downstream call, it multiplies again with every extra cluster, and retries multiply it once more.
Each downstream call pays the proxy time once, and each retry is charged per call.
Average app time per call is 35 ms plus 0.25 ms per percent of canary weight (stable 35 ms, canary 60 ms).
Each retry costs 8 ms per downstream call.
Each extra cluster adds 32 ms per call.
Try this: (1) Leave the defaults: 2 calls, canary 10 percent, 1 retry, 1 cluster. The total is about 143 ms. (2) Push the canary to 100 percent, the retries to 3, the calls to 8, and the clusters to 3. The total is about 1,392 ms. Same services, same code. Only the knobs changed. (3) Set everything back, then set the clusters to 3 and watch the total jump by 128 ms. That is what two extra clusters cost on two calls, and it is why multicluster is a decision, not a default.
Three questions. No score is saved anywhere — this is just you against the material.
Linkerd looks like one small mesh. It is not. It is four layers stacked on each other, and each layer only makes sense after the one below it. I learned it in this order, and I teach it in this order: get a proxy running and encrypt one call, then shape the traffic, then lock it down and watch it, then link clusters and debug the quiet failures. Skip a layer and the next one reads like magic. Build them in order and nothing in this lesson is magic.
Definition: Linkerd at the basic level is a proxy injected next to each app container, plus a control plane that tells the proxies what to do. The proxy is written in Rust, so it is small and fast. The app makes a plain call by service name. The proxy intercepts it, encrypts it with mTLS, and sends it to the target proxy. mTLS is on by default, with no permissive mode to forget.
Types:
Uses: when a team has a few services and wants encryption and observability without a big mesh. One install, one annotation, and every call is encrypted and recorded.
Dimensions: Scale: holds up to a few dozen services without special care. Failure modes: a pod without a proxy is outside the mesh and no rule touches it, so a missing injection annotation is the most common basic failure; a proxy that cannot reach the control plane runs on stale config, and the stale config is the one that breaks. Cost: one proxy per pod, a small slice of CPU and memory per pod, smaller than Istio's Envoy, paid on every pod whether it needs the mesh or not. Trade-offs: you pay the proxy tax on every call before you get any of the advanced features, so a service that never needs routing, policy, or telemetry is paying for a mesh it does not use. The tax is smaller than Istio's, and that is the trade Linkerd makes.
Definition: the intermediate layer is traffic shaping with TrafficSplit and the proxy's own retry and timeout settings. A TrafficSplit says how much traffic goes to each version of a service, by weight, so 90 percent goes to the stable version and 10 percent to the canary. Retries say how many times a failed call is tried again. Timeouts say how long a call may take before it is cut off.
Types:
Uses: safe deploys and honest tests. A canary gets 10 percent of real traffic before it gets all of it. A retry is allowed only on routes the service profile marks as safe, because retrying a payment is not the same as retrying a read.
Dimensions: Scale: holds to a few hundred services, but the split count grows fast, because every canary is a split the control plane has to push. Failure modes: a retry policy with no timeout turns one slow call into several slow calls; a split that sends traffic to a version with no endpoints drops every call it sends; a service profile that marks a write as retryable retries a payment, and the payment goes through twice. Cost: config, not compute; each rule is cheap to run, and the bill is the time it takes a human to read all of them when something breaks. Trade-offs: every knob in this layer can hurt as easily as it helps, so change one knob at a time and watch the trace. A canary and a retry policy changed together is two variables, and two variables cannot be debugged.
Definition: the advanced layer is identity, permission, proof, and the second cluster. A ServerAuthorization decides which identity may call which endpoint. The observability stack records every call, its duration, and its result, and the dashboards show the golden metrics: request rate, success rate, and latency, per service, with no app code. Multicluster links two clusters, so a service in one can call a service in the other, through a gateway, with the same mTLS and the same policy.
Types:
Uses: when the mesh has to prove something, not just move traffic. A payments service that must only answer checkout. A dashboard that shows the p99 for every service without any app code. Two clusters that must call each other without opening the whole network.
Dimensions: Scale: the control plane becomes the bottleneck; watch push latency and the share of proxies on the latest config. Failure modes: a policy that allows a whole namespace when it meant to allow one service; a multicluster link with the wrong trust anchor fails handshakes exactly like an expired certificate; a dashboard that shows golden metrics for a service with no traffic is a dashboard that says nothing, and the silence looks like health. Cost: operational; certificates rotate, policies need owners, and the metrics have to go somewhere and be read by someone. Trade-offs: mTLS on by default buys verified identity and costs nothing to turn on, because there is nothing to turn on. Multicluster buys one mesh across regions and costs a road that is paid on every call. Observability buys the truth about every call and costs the storage to keep it.
Tap: watch the live requests. Linkerd tap shows live requests as they pass through a proxy, in real time. It is the fastest way to see whether a service is actually receiving traffic, and what the traffic looks like. I tap before I guess, because a tap shows the truth and a guess shows my assumptions.
Do this: run the tap command from the snippet against a service, and send a request. Expected: the request appears in the tap output, with the source, the destination, and the result. Send a request to a service that should not receive it, and expect nothing in the tap. The tap is the evidence, and the absence is the evidence too.
linkerd viz tap deployment/shop-api
# send a request, and expect it to appear in the tap output
What the snippet means: tap streams the requests the proxy sees. A request that appears is a request that reached the service. A request that does not appear never reached it, and the tap is the proof.
Read the flags: linkerd viz tap
Linkerd is quiet, and quiet is hard to debug. Tap shows live requests as they pass through the proxy, with their result codes. A tap that shows nothing is a tap that found no traffic, and that is a finding too.
linkerd viz tap: This shows live requests through a resource proxy. Each line is one request with its result code. It runs until I stop it with Ctrl-C. Without traffic, it prints nothing, which tells me the service is not receiving calls.deployment/shop-api: This names the resource: a Deployment called shop-api. Tap needs the type and the name, because the mesh holds many kinds of objects.-n (added): The lesson did not show this one yet. It picks the namespace, like linkerd viz tap deployment/shop-api -n shop. Without it, tap uses the current namespace, which may not hold the deployment.Try it: run linkerd viz tap deployment/shop-api, then send a request. Expect the request to appear in the tap output with its result code. Send no request and expect silence.
The Server that the policy forgot. A Server names the port a policy applies to, and a ServerAuthorization decides which identity may call it. A port with no Server has no policy, and a policy with no Server applies to nothing. I list the Servers when a policy does not seem to work, because the missing Server is the usual cause.
Do this: create a ServerAuthorization for a port, then list the Servers with the command in the snippet. Expected: a Server that names the port, or nothing. If there is nothing, create the Server, and expect the policy to start working. The Server is the scope, and the policy without a scope is a rule with nowhere to apply.
kubectl get server -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,PORT:.spec.port
kubectl get serverauthorization -A
What the snippet means: the two lists are the scope and the rule. A policy without a Server is a rule that applies to nothing, and the Server list is the check.
Read the flags: kubectl get server -A -o custom-columns and kubectl get serverauthorization -A
A Server names a port and its policy. A ServerAuthorization says who may call it. Get lists both, so I can see the policy and the permission side by side.
kubectl get server -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,PORT:.spec.port: The -A flag lists Servers across all namespaces. The -o flag sets the output format. Custom-columns prints the namespace, the name, and the port as columns. The part before each colon is the header. The part after is the JSON path.kubectl get serverauthorization -A: This lists the ServerAuthorizations across all namespaces. Each line is one permission. Without a format flag, get prints the default table, which is enough to see the names.--show-labels (added): The lesson did not show this one yet. It adds the labels as the last column. I use it when I want to see which Server a ServerAuthorization selects, because the selection is by label.Try it: run kubectl get server -A -o custom-columns and read the port of each Server. Run kubectl get serverauthorization -A next and check that each Server has a permission that selects it.
Definition: the master layer is debugging a mesh that is too quiet to tell you what is wrong. Linkerd's tap command shows live requests as they pass through a proxy, in real time. The destination controller is the source of truth for endpoints, and when it is slow, the proxies wait. The contrast with Istio is the design choice underneath it all: Istio is a platform with a filter chain, Wasm plugins, and an ambient mode. Linkerd is a mesh that does less, on purpose, and the things it leaves out are the things you do not have to debug.
Types:
Uses: when Linkerd stops telling the truth: when the config says one thing and the traffic does another, and the dashboards are too quiet to show why. I tap the proxy and watch the live requests. I check the destination controller and ask whether it is behind.
Dimensions: Scale: holds at thousands of services, because the proxies are small and the control plane is small, and the cost moves from per-pod proxies to the destination controller's watch. Failure modes: a destination controller that is slow hands out stale endpoints, and the proxies wait for it; a tap that shows no requests is a proxy that was never injected, and the silence is the symptom; a team that picks Linkerd and then needs a Wasm plugin finds that Linkerd does not have one, and the need becomes a migration, not a config change. Cost: expertise; debugging a quiet mesh needs a team that can read a tap and a controller log. Trade-offs: Linkerd buys simplicity and costs capability. Istio buys capability and costs complexity. A mesh that does less is a mesh that breaks less, and a mesh that does more is a mesh that can do the job Linkerd cannot. The choice is the trade-off, and the trade-off is the lesson.
The destination controller is slow. The destination controller is the source of truth for endpoints, and when it is slow, the proxies wait. A service that is slow to appear in the mesh is usually a destination controller that is slow to answer, and the fix is the controller, not the service.
Do this: check the destination controller logs with the command in the snippet, and look for slow responses. Expected: a log line with the time each lookup took, or silence. Deploy a new service, and watch how long it takes to appear in the mesh. If it takes minutes, the controller is the bottleneck, and the logs will say so.
linkerd check --proxy
kubectl logs -n linkerd deployment/linkerd-destination | tail -20
What the snippet means: the check reports the health of the install, and the logs show the controller at work. A controller that is slow makes every proxy slow to learn about new endpoints.
Read the flags: linkerd check --proxy and kubectl logs -n
Check is the health check for the whole install. The --proxy flag checks the data plane proxies too, not just the control plane. Logs from the destination controller show what the control plane did about the proxies it found.
linkerd check: This checks the control plane, the data plane, and the extensions, and reports each check as pass or fail. I run it after every change to the mesh. A fail names the check and the fix.linkerd viz check: This checks the viz extension only. The lesson shows it next to check. A viz check that fails while check passes means the dashboard is broken, not the mesh.--proxy: This checks the proxies too. Without it, check looks at the control plane only, and a broken proxy hides. With it, check reports each proxy and whether it is running the right version.kubectl logs -n linkerd deployment/linkerd-destination: This prints the logs of the destination controller. The -n flag picks the linkerd namespace. The controller resolves service names to pod IPs, so its logs show the discovery the proxies depend on.| tail -20: The pipe keeps the last twenty lines. The full log is long, and the last twenty lines are the recent past.Try it: run linkerd check --proxy, then kubectl logs -n linkerd deployment/linkerd-destination | tail -20. Expect check to pass and the logs to show the destination controller answering lookups.
Multicluster: the service in the other cluster. Multicluster links two clusters, so a service in one can call a service in the other, through a gateway, with the same mTLS and the same policy. The mirror makes the service in the other cluster visible in this one, and the call looks like a local call. I test it with a call, because a link that is not tested is a link that is assumed.
Do this: list the mirrored services with the command in the snippet, then call one from a pod in this cluster. Expected: the call succeeds, and the response comes from the other cluster. Remove the link, call again, and expect the call to fail. The failure is the link working, and the test from both sides is the proof.
kubectl get service -A -l mirror.linkerd.io/mirrored-service
kubectl exec -it my-pod -- curl -s http://shop-api-remote/api -o /dev/null -w "%{http_code}\n"
What the snippet means: the first command lists the mirrored services, and the second calls one. A mirrored service that answers is a link that works. A mirrored service that does not answer is a link that is broken.
Read the flags: kubectl get service -A -l and kubectl exec with curl
A mirrored service is a service from the other cluster, copied into this one. The label picks only the mirrored ones. Curl from inside a pod proves the mirror answers.
kubectl get service -A: This lists the Services across all namespaces. The -A flag is the short form of --all-namespaces. Without it, get shows only the current namespace.-l mirror.linkerd.io/mirrored-service: The -l flag picks services by label. Here the label has a key and no value, so it matches every service that carries the label. Those are the mirrored services.kubectl exec -it my-pod -- curl -s http://shop-api-remote/api -o /dev/null -w "%{http_code}\n": Exec runs the curl inside the pod my-pod. The -it pair gives a terminal. The double dash separates kubectl flags from the curl. The curl asks the remote service for its API and prints only the status code.-o wide (added): The lesson did not show this one yet. It adds node and IP columns to the table, like kubectl get service -o wide. I use it when the question is where a service points, not just whether it exists.Try it: run kubectl get service -A -l mirror.linkerd.io/mirrored-service and read the mirrored services. Run the exec curl next and expect 200 from the remote service.
Istio and Linkerd answer the same question two ways. Istio is a platform. It has a filter chain, Wasm plugins, an ambient mode, and a config for everything. Linkerd is a mesh. It has a small proxy, mTLS on by default, traffic splits, and policy, and it leaves the rest out on purpose. I pick Istio when the team needs the platform: custom filters, complex routing, a mesh that can be extended. I pick Linkerd when the team needs the mesh: encryption, observability, and traffic splitting, with a control plane a small team can run. The wrong move is picking the big one for a small team, because the big one needs a team to run it, and the wrong move in the other direction is picking the small one and then needing a Wasm plugin, because Linkerd does not have one. The choice is the trade-off, and the trade-off is the lesson.
The proxy is written in Rust, and the language is the whole reason the proxy is small. Rust has no garbage collector and no runtime, so the proxy starts fast and uses little memory. The proxy is a single binary, and it does one job: move requests, encrypt them, and record them. Istio's Envoy is written in C++ and does more, and the more is the reason it is bigger. I do not pick a mesh by language. I pick it by what the language buys. Rust buys a proxy that is cheap to run on every pod, and cheap matters when there are a thousand pods.
Multicluster in Linkerd is a link between two clusters. The link is a gateway in each cluster, and the gateway carries the traffic between them. The trust anchor is the shared root certificate, and each cluster has to hold the other's anchor, or the handshake fails. The link is the road, and the road is paid on every call, in both directions. I link clusters when a service genuinely lives in another cluster, not when a team wants a second cluster for fun, because the link is a dependency, and a dependency that is down is a service that cannot be reached. And I test the link by calling across it, because a link that was never used is a link nobody knows works.
Linkerd is quiet, and quiet is hard to debug. When a call fails, I tap the proxy. The tap command shows live requests as they pass through, with their result codes, and a tap that shows no requests is a proxy that was never injected. I run linkerd check, because it checks the whole install: the control plane, the proxies, the certificates, and the webhooks, and it names the thing that is broken. I check the destination controller, because it is the source of truth for endpoints, and a controller that is slow hands out stale endpoints. The mesh that tells you nothing is the mesh you have to ask directly, and the tap is how you ask.
At 10x the services, the problem is the destination controller. Every proxy asks it for endpoints, and the controller has to answer them all. I watch its latency and its memory, and I split the mesh before the controller becomes the bottleneck. At 100x, the design changes. The mesh stops being one mesh and becomes federated meshes, one per cluster, with explicit links between them. The policy moves from per-service rules to team-owned defaults with platform guardrails. And the mesh tax gets a budget line of its own, because at that scale the tax is a real slice of p99, and a tax nobody budgets for is a tax nobody controls. The small proxy stays small at 100x. That is the whole reason it is small.
apiVersion: split.smi-spec.io/v1alpha2
kind: TrafficSplit
metadata:
name: pricing
namespace: shop
spec:
service: pricing
backends:
- service: pricing-stable
weight: 900m
- service: pricing-canary
weight: 100m
---
apiVersion: policy.linkerd.io/v1beta1
kind: Server
metadata:
name: pricing
namespace: shop
spec:
podSelector:
matchLabels:
app: pricing
port: 8080
---
apiVersion: policy.linkerd.io/v1beta1
kind: ServerAuthorization
metadata:
name: pricing-allow-checkout
namespace: shop
spec:
server:
name: pricing
client:
meshTLS:
serviceAccounts:
- name: checkout
namespace: shop
---
apiVersion: v1
kind: Namespace
metadata:
name: checkout
annotations:
linkerd.io/inject: enabled
Four pieces, one Linkerd story. The TrafficSplit sends 90 percent of pricing traffic to stable and 10 percent to canary, using milli-weights: 900m and 100m add up to 1000m, which is the whole. The Server names the port the policy applies to: the pricing pods, on port 8080. The ServerAuthorization says only the checkout service account may call that server, and it checks the mTLS identity, not the IP address. The namespace annotation is the injection: every pod in the checkout namespace gets a proxy. Read them together and you can see the whole trip: split, name, allow, inject. Change one of them without reading the other three and you have changed the trip in a way you did not intend.
linkerd check is the health check for the whole install, and I run it after every change to the mesh. It checks the control plane, the data plane, and the extensions, and it reports the failures in plain words. A check that passes is an install that works, and a check that fails names the thing to fix.
Do this: run the check from the snippet, and read the output. Expected: a list of checks, all passing, or a failure that names the component. Fix the first failure, run again, and expect the list to shrink by one. The check is the test, and the test runs after every change.
linkerd check
linkerd viz check
What the snippet means: the two checks cover the mesh and the observability extension. A failure names the component, and the fix is the component, not the check.
This one stayed with me because the mesh was quiet and the dashboards were green, and the users were waiting. The proxy was fine. The controller behind it was slow, and nobody was watching the controller.
A team ran Linkerd across two clusters. A checkout service in the first cluster called a pricing service in the second, through the multicluster link. The destination controller in the first cluster fell behind on its watch, after a deploy added a hundred new services at once. The controller handed out endpoint lists that were minutes old, and the proxies kept sending calls to pricing pods that had already been replaced. The calls timed out, the retries fired, and the retries landed on the same dead endpoints. The dashboards stayed green, because the proxies recorded the calls as attempted, and the app dashboards stayed green, because pricing was healthy in its own cluster. The on-call team checked the app, then the mesh, then the link, and found nothing wrong at every hop, because the link was up and the app was healthy. The controller was the only thing that was slow, and nobody was watching it.
Three things. First, the destination controller gets its own dashboard and its own alerts, because a mesh is only as fast as the answers its control plane gives, and nobody was watching the control plane. Second, the controller's memory is sized for the service count, with headroom, because a controller at its memory limit is a controller that falls behind. Third, the deploy that adds a hundred services at once is split into batches, because a controller that has to learn a hundred new services at once is a controller that falls behind while it learns. The mesh did nothing wrong in this incident. The lesson is that the control plane is a service too, and a service nobody watches is a service that fails quietly.
Design the Linkerd setup for a platform with 60 services across 2 clusters. Checkout calls pricing, fraud, ledger, and notifications. Fraud must never be callable from outside its namespace. Deploys must canary at 10 percent and roll back on their own when latency climbs. mTLS is on by default. One external payment provider sits outside the mesh. The latency budget is 300 ms at p99 for a checkout request. Sketch the install, the traffic splits, the policies, and the multicluster link, and say what you would measure. Then say what would change if the team picked Istio instead.
Start with the budget from this lesson. One checkout request takes about 61 ms inside one cluster: fixed proxy hops about 26 ms, the app about 35 ms. Four downstream calls in sequence would be about 244 ms, which is inside the 300 ms budget but leaves little headroom, so I run pricing and fraud in parallel, not in sequence, and I move ledger and notifications after the response, so the user never waits for them. The parallel pair costs about 61 ms plus the slower of the two apps, roughly 100 ms end to end, which leaves about 200 ms of headroom. Little's Law for the sizing: at 500 checkout requests a second, each spending 0.061 seconds inside the mesh path, the number inside the system at once is 500 times 0.061, which is about 31 concurrent requests. At a peak of 1,500 requests a second it is 1,500 times 0.061, which is about 92 concurrent requests. I size the destination controller and the proxy connection pools for 92 concurrent, plus headroom, and I alert at 80 percent of the pool. A multicluster call takes about 105 ms instead of 61, so the same 500 requests a second across clusters is 500 times 0.105, which is about 53 concurrent, and the gateway pool for the cross-cluster road is sized for that number, not for the single-cluster one. Against Istio, the same request cost about 76 ms in Lesson 91, so Linkerd saves about 15 ms per call on this trip, and the saving is the smaller proxy.
Linkerd buys simplicity and costs capability. Istio buys capability and costs complexity. A TrafficSplit buys a canary and costs a split the control plane has to push. mTLS on by default buys verified identity and costs nothing to turn on, because there is nothing to turn on. Multicluster buys one mesh across regions and costs a road that is paid on every call. The destination controller buys a single source of truth and costs a single place that can fall behind. And the target is p99, not the average. An average of 100 ms with a p99 of 400 ms fails this design, and the p99 is where the slow controller and the dead endpoints live.