Platform Engineering · Lesson 161

Kubernetes Lab: Networking and Troubleshooting

The networking and ingress lessons taught you Services, Ingress, and NetworkPolicies. This lab is where you use them, and where you learn the troubleshooting order: logs, describe, exec. You expose the app with an Ingress, you lock it down with a NetworkPolicy, you break a pod on purpose, and you find the cause in the logs before you fix it. By the end you have an app that answers through its own name, and a broken pod you fixed by reading, not by guessing.

Difficulty: Advanced · Know this first: Lesson 48 — Kubernetes Core; Lab 160; minikube or k3s installed, kubectl installed, an ingress controller installed

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario — not a measurement from a live system — but the shape is honest: on a cache hit the trip is almost free, on a miss the kitchen eats the budget. Step through the lesson or switch the scenario and watch the bar change.

Extra diagram · from scratch to master

The networking lab, mapped: service, ingress, policy, fix

Start with one Service. Add one box at a time until the app answers by name and the broken pod is fixed by reading.

7 boxes on the trip1 Ingress1 CrashLoop
Cluster
0 min
Service
1 min
Ingress
3 min
Policy
1 min
Break
1 min
Logs
1 min
Crash
trip ends
Request / flowResponse · the way backFailure / dead end

Try it: change the inputs and watch the trip bend

The budget bar above shows three fixed stories. This simulator lets you write your own. Move a slider and watch the routing, the debugging, and the total change in the budget bar, in the hop timings, and in the total. The numbers are a teaching example, not live measurements. The shape is the lesson: more apps means more routing, and a crash costs a read of the logs.

1

How many apps have their own Ingress. Each app is one more host, one more route, and one more name to check.

0

How many pods are in CrashLoopBackOff. Each one needs its logs read before it can be fixed.

1

How many NetworkPolicies lock the apps down. Each one is one more rule to check when a connection times out.

Try this: (1) Set apps to 1, crashes to 0, policies to 1. The route is about 4 min, the debug about 1 min, the prove about 1 min, and the total is about 11 min. (2) Set all three to the worst end: 5 apps, 5 crashes, 5 policies. The route is about 20 min, the debug about 11 min, the prove about 1 min, and the total is about 37 min. Same cluster. Only the inputs changed.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Try it: edit the pipeline and check it

Here is the Ingress and the NetworkPolicy from this lab, ready to edit. Change it, then run the check. The check looks for the five things a working pair needs: kind Ingress, a host rule, a Service backend, kind NetworkPolicy, and an ingress rule.

Try breaking it on purpose: change the host to the wrong name, or delete the ingress rule from the policy. Run the check and watch it fail. Then press Reset and try a change that keeps it passing, like adding a second path.

Press "Run the check" to see whether this pipeline does what the lesson asks.

Expected output when it passes: The file has an Ingress routing hello.example.test to the hello Service, and a NetworkPolicy allowing only the web. kubectl apply would create both, and the app would answer by name.

kubectl apply -f, flag by flag. apply reads a file and moves the cluster toward it. The file is the desired state. The flag says which file.

  • -f deploy.yaml names the file to apply. Without it the command has nothing to read. The file is the desired state, and apply is what moves the cluster toward it.

Flags you will meet next: --dry-run=client prints what would happen without doing it, and -n picks the namespace.

Try it: apply the same file twice. The second run prints unchanged, not created. That is the proof apply is declarative: it compares, and it only changes what differs.

Basic to master

I take the same app four times here. First I address it. Then I route it. Then I lock it. Last I break it and fix it.

Basic — Three Service types

Definition: ClusterIP is reachable inside the cluster only. NodePort opens a port on every node. LoadBalancer asks the cloud for a public IP. The Ingress is how a ClusterIP app answers the outside world without a public IP per Service.

Types:

  • ClusterIP, which is inside only.
  • NodePort, which opens a port on every node.
  • LoadBalancer, which asks the cloud for a public IP.
  • An Ingress, which routes by name to a ClusterIP Service.

Uses: I use ClusterIP for everything inside the cluster, and an Ingress for the outside. Try it now: patch the lab Service to NodePort, read the port, and patch it back.

Dimensions: At low volume everything is instant. What breaks first is a NodePort left open: a lab that changes the cluster and does not put it back.

I stop here on purpose. The three types above are the front of the story. Behind a Service sit EndpointSlice objects, which list the ready pods, and CoreDNS, which turns the Service name into its ClusterIP. I cover neither in this lab. Lesson 105, Kubernetes Networking and CNI covers both in its new Service internals section. Read that section next if you want the full path from name to pod.

Try it: Basic

My app runs behind a ClusterIP Service. How does it answer the outside world without a public IP per Service?

Now do it: On your cluster, run the Basic drill above. Patch the lab Service to NodePort, read the port, and patch it back.

Intermediate — Ingress routes by name

Definition: An Ingress routes by host and by path. The host is the name, the path is the route, and the backend is the Service. The controller watches the Ingress files and configures the routing. A /etc/hosts line makes a test name resolve to the cluster on your laptop.

Types:

  • A host rule, which names the app.
  • A path, which is the route.
  • A backend Service, which is the destination.
  • A controller, which is the thing that routes.

Uses: I use an Ingress whenever an app needs a name, which is every app users reach. Try it now: curl the lab app by name and check it answers.

Dimensions: As the apps grow, the hosts multiply, and the routes multiply with them. What breaks is a host that does not match: the Ingress exists, and the name goes nowhere.

Try it: Intermediate

My Ingress exists but the name goes nowhere. What do I check first?

Now do it: On your cluster, run the Intermediate drill above. Curl the lab app by name and check it answers.

Advanced — NetworkPolicy is a firewall in labels

Definition: A NetworkPolicy selects pods by label and allows only the traffic it names. The default is allow all, until a policy selects a pod, and then the default becomes deny all except what the policy allows. A policy block looks like a timeout, not a refusal.

Types:

  • A podSelector, which picks the pods the policy covers.
  • An ingress rule, which names who may call them.
  • A port, which names what may be reached.
  • A timeout, which is the first clue a policy is blocking.

Uses: I use a NetworkPolicy whenever an app must talk to only one caller, which is most apps. Try it now: apply the lab policy and test it from a pod with and without the web label.

Dimensions: At ten times the policies, the rules multiply, and the labels matter more. What breaks is a label that does not match: the policy exists, and the pod it was meant for is not covered.

Try it: Advanced

I applied a NetworkPolicy and the call now times out instead of failing fast. What does the timeout tell me?

Now do it: On your cluster, run the Advanced drill above. Apply the lab policy and test it from a pod with and without the web label.

Master — Logs first, describe second, exec third

Definition: At this level troubleshooting is an order, not a guess. Logs first, because the app knows why it crashed. Describe second, because the cluster knows what happened around the crash. Exec third, because exec is for looking inside a running pod, and a crashing pod is not running long enough to look inside.

Types:

  • kubectl logs, which is first.
  • kubectl logs, flag by flag. Logs are the first thing I read when a pod misbehaves. The label flag reads them from every matching pod at once.

    • -l app=hello reads logs from every pod that matches the label. Without it logs needs one pod name, and the failing pod may not be the one I picked.
    • --tail=20 shows only the last 20 lines from each pod. Without it logs prints the whole log from the start.

    Flags you will meet next: -f follows the log, --since 10m shows the last ten minutes, and -c container picks one container in the pod.

    Try it: run it with --tail=20, then without it. The first gives you the last twenty lines. The second gives you the whole log, which on a busy pod is pages of old news.

  • kubectl describe, which is second.
  • kubectl describe, flag by flag. describe is the command I run when something is wrong. The events at the bottom are where the reason lives.

    • describe pod hello-pod prints the events and status of one object. The events are where the reason for a failure lives: the image that would not pull, the port that was already taken.
    • describe pod -l app=hello describes every pod that matches the label. Without the label I have to name each pod, and I may miss the one that failed.

    Flags you will meet next: -n picks the namespace, and kubectl get events --sort-by=.lastTimestamp lists the newest events first.

    Try it: describe a healthy pod, then describe a failing one, and read the Events block in both. The healthy one has clean pulls and starts. The failing one names the image that would not pull or the port that was taken.

  • kubectl exec, which is third.
  • kubectl exec, flag by flag. exec runs a command inside a running container. The -- separator is the flag that makes it work at all.

    • deploy/hello names the workload. kubectl picks a pod from it. Without a target, exec has nowhere to run.
    • -- marks the end of kubectl flags. Everything after it is the command to run inside the container. Without it kubectl tries to read the container command as its own flag, and the command fails before it starts.
    • cat /data/marker.txt the command that runs inside the container. It reads a file and prints it. If the file is missing, the error is the answer: the data did not survive, or it was never written.

    Flags you will meet next: -it gives you a terminal inside the container, -c container picks one container in a multi-container pod, and -- sh -c '...' runs a compound command.

    Try it: run the cat command, then run it again after writing the marker file. The first read fails or prints nothing. The second prints the marker. The file is the proof the volume survived, and exec is how I checked it.

  • A curl, which is the proof the fix worked.

Uses: I use this order for every broken pod, every time, without skipping a step. Try it now: break the lab pod, read the logs, and fix the one number they name.

Dimensions: At a hundred pods, the order is the same, and the logs are the same. What breaks is guessing: a fix applied without reading the logs fixes the wrong thing, and the pod crashes again.

Try it: Master

My pod keeps crashing. What do I read first, and why?

Now do it: On your cluster, run the Master drill above. Break the lab pod, read the logs with --tail=20, and fix the one number they name.

Go deeper

The Ingress and the NetworkPolicy, the whole file. Do this: save it as ingress.yaml and run kubectl apply -f ingress.yaml. Expected output: ingress.networking.k8s.io/hello created and networkpolicy.networking.k8s.io/hello-allow-web-only created. This file was checked in this lab: it parses clean with yaml.safe_load, it holds two documents, and the Ingress routes hello.example.test to the hello Service.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
    - host: hello.example.test
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service: {name: hello, port: {number: 80}}
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: hello-allow-web-only
spec:
  podSelector:
    matchLabels: {app: hello}
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector: {matchLabels: {app: web}}
      ports: [{port: 80}]

The troubleshooting order, the whole method. Do this: run kubectl logs -l app=hello --tail=20 first, then kubectl describe pod -l app=hello second, then kubectl exec deploy/hello -- cat /etc/nginx/nginx.conf third, only if the pod is running. Expected output: the logs name the cause on the first line, the describe shows the crash and the back-off in Events, and the exec shows the config the pod is running with. If the pod is crashing, skip exec: it is not running long enough to look inside.

What CrashLoopBackOff means: the container starts, crashes, and Kubernetes restarts it, again and again, with a growing wait between tries. Do this: run kubectl get pods and read the RESTARTS column. Expected output: the count climbs, 1, 2, 3, and the STATUS reads CrashLoopBackOff. The count is the number of crashes, and the status is the sign that says to read the logs.

kubectl get, flag by flag. get lists objects. The label flag is how I ask for the three pods I care about instead of every pod in the cluster.

  • -l app=web filters by label. Without it I get every pod in the namespace, and the three I care about hide in the list.
  • -o yaml prints the whole object as YAML instead of the short table. Without it I see a summary, and the field I need is not in it.

Flags you will meet next: -n picks the namespace, -o wide adds the node and IP columns, and --all-namespaces lists every namespace at once.

Try it: run kubectl get pods, then kubectl get pods -l app=web. The first lists everything. The second lists only the web pods. Then add -o yaml to one pod and read the full object.

Honest limit, stated plainly: this lab was verified by parsing the manifests and by reading the kubectl docs, not by running kubectl apply. There is no cluster in the machine this lab was written on, so no pod was actually started here. The commands, the outputs, and the checks are the ones the reader runs on their own machine, where minikube or k3s is running and an ingress controller is installed.

At 10x the apps, the shape holds and the Ingresses multiply. At 100x, the design changes: the routing moves into a gateway, and the policies move into a mesh. The order never changes, though. Logs first, describe second, exec third.

This lab practices Lesson 48 — Kubernetes Core: Services, the objects this lab exposes and locks down; and Lesson 105 — Kubernetes Networking & CNI: the NetworkPolicy this lab locks the app down with; and Lesson 44 — Kubernetes Ingress: the Ingress this lab exposes the app through.

One time it broke

What broke

A pod I owned crashed on startup, and the team spent an hour guessing: the image, the config, the node, the network. Someone finally read the logs, and the first line said the app had tried to listen on a port that was already in use. The fix was one number, and it took a minute. The hour before it was spent on everything except the logs, because the logs felt too simple to be the answer.

Timeline

  • 10:00,the pod crashed on startup and went into CrashLoopBackOff.
  • 10:10,the team checked the image, the config, and the node. All three were fine.
  • 10:30,someone suggested the network, and the team checked the Service and the Ingress. Both were fine.
  • 11:00,someone finally ran kubectl logs. The first line said address already in use.
  • 11:05,the port was changed, the pod started, and the app answered. The fix took a minute. The guessing took an hour.

What changed after

Two things. First, the troubleshooting order is written down and taped to the wall: logs, describe, exec. Second, the first line of the log is read before any other tool is opened. Try it now: break the lab pod, read the logs, and time how long the fix takes when you start with the logs.

If you remember nothing else

Build it yourself

Build an Ingress and a NetworkPolicy for an app you choose, break a pod on purpose, and fix it by reading the logs. It passes when the app answers by name, the policy allows only the right caller, and the broken pod is fixed with a one-line change.

Build it, step by step

  • Expose your app with an Ingress on a test host, add the /etc/hosts line, and curl it by name.
  • Write a NetworkPolicy that allows only one caller label, apply it, and test it from a pod with and without the label.
  • Break a pod on purpose by changing its port to one that is already in use, and confirm it reads CrashLoopBackOff.
  • Read the logs, name the cause, and fix the one number that is wrong.
  • Apply the fix, curl the app by name again, and confirm it answers.

What success looks like

Success looks like this: curl by name answers, the policy blocks the wrong label and allows the right one, the broken pod reads CrashLoopBackOff, the logs name the cause, and the fixed pod answers again. If the curl fails after the fix, the Ingress host is wrong. If the policy blocks everyone, the label is wrong.

If it fails, check this

If the Ingress has no address, the controller is missing. If curl by name fails, check the /etc/hosts line first. If the policy blocks the right caller, check the label on the calling pod. If the logs are empty, the pod crashed before it logged anything, and describe is the next step. If exec fails, the pod is not running long enough to look inside.

Next in this series
Lesson 49 — Infrastructure as Code