Platform Engineering · Lesson 172

Load Balancers in the Cloud: Internal vs External, ALB vs NLB vs GWLB

Lesson 06 taught me what a load balancer does. One box in front of several servers, deciding request by request who does the work. The cloud does not change that job. It changes where the box lives, who owns it, and how many kinds of it there are. I have stood in front of a whiteboard with two services that looked the same and needed opposite answers. One was a shop that had to face the internet. The other was a billing worker that had to stay private. Same traffic shape. Different load balancer, different scheme, different failure modes. This lesson walks one request through the cloud version of the balancer, hop by hop, and shows the three kinds you will actually meet: ALB, NLB, and GWLB. And it shows the one scenario that still catches teams: a health check that lies.

Difficulty: Advanced · Know this first: Lesson 06, Lesson 01, Lesson 44

The story: two services, opposite answers

A fresher on my team got two services to put behind load balancers. The first was the shop. It had to face the internet. Anyone in the world could open it. The second was the billing worker. It charged cards. It had to stay private. Only services inside our network could call it, and nobody outside could even find it.

She came to me with one question. Same traffic shape, same number of servers. Can I use the same load balancer for both? The answer was no, and the reason is this lesson. The shop needs an internet-facing balancer with a public name. The billing worker needs an internal balancer with a private name and no public address at all. Same job, opposite scheme. Pick the wrong one and the billing worker is on the internet, or the shop is unreachable. Both mistakes are one field in the config. One word. internal = true or internal = false.

Then she asked the second question. Which kind? ALB, NLB, or GWLB. The shop speaks HTTP, so it gets an ALB. The ALB reads the host and the path and routes. The billing worker speaks HTTP too, so it also gets an ALB, just an internal one. The firewall appliance the security team runs is not a server at all. It inspects packets. That one gets a GWLB, because the GWLB exists to put an appliance in the path. Three kinds, three jobs. This lesson is the map.

What you will build

By the end of this lesson you will have built, in Terraform, an internet-facing ALB for a shop with a target group, a deep health check, and an HTTPS listener carrying two certificates through SNI. Then you will build the same shape as an internal ALB for a private billing worker, and again as an NLB. You will prove SNI with two hostnames on one port, on your own machine, with no cloud account. And you will be able to answer the two questions that decide every cloud load balancer: internal or internet-facing, and ALB, NLB, or GWLB.

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one checkout, not a measurement from a live system, but the shape is honest: on the normal path, the target does the real work and the balancer's hops are nearly free. DNS is the biggest slice, and it is the only one you cannot tune at the balancer. Step through the lesson or switch the scenario and watch the bar change. The internal path is faster because DNS answers inside the VPC. The broken path costs almost the same as the normal one, because the failure is cheap in time. The expensive part is that the balancer trusted the check.

Internal vs external, side by side

Same balancer, same target group shape, opposite exposure. The left panel is the shop. Anyone on the internet can reach it, because that is the job. The right panel is the billing worker. Only callers inside the VPC can reach it, because that is the job. The difference is one field in the config, and it decides the whole security story. Read the two panels left to right, then bottom to top: the bottom line is the one that matters.

Internet-facing ALB shop-alb · internal = false Internet anyone ALB listener :443 TLS ends here target 1 :8080 target 2 :8080 public DNS name anyone on the internet can reach it that is the job, and the risk Internal ALB billing-alb · internal = true caller in the VPC ALB listener :80 target 1 :8080 target 2 :8080 private DNS name only no public IP · the name does not resolve outside the VPC

The two panels share everything except the scheme. Same listener shape, same target group shape, same health check. The left one has a public name and a public risk. The right one has no public anything. When a service must stay private, I do not add a firewall rule and hope. I build it internal, and then there is nothing to reach.

How many requests hit a sick target?

The budget bar above shows three fixed stories. This simulator lets you write your own. It answers one question: when a target goes sick, how many requests fail before anyone notices. You set how many requests arrive per second, how many targets there are, how many of them are sick, and whether the health check catches them. Move a slider and watch the pool, the per-target load, the sick share, and the verdict change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: a lying check does not add load. It adds failures, at the same load.

600 req/s

How many requests arrive every second. Six hundred is a busy shop at noon. Five thousand is a peak. More arrivals means more load on every target in the pool, healthy or sick.

6 targets

How many servers the target group holds. Two is a small service. Twelve is a real pool. More targets means less load each, and a smaller share for any one sick target.

1 sick

How many targets lost their database connection, or broke in a way the check cannot see. Zero is a healthy pool. Five is a bad deploy. A sick target can never be more than the pool minus one: one healthy target must stay.

Checked means the check is deep: it asks the database, so a sick target leaves the pool and takes zero requests. Unchecked means the check only asks the process, so the sick target stays in rotation and takes its fair share. That is the Lesson 06 lie, with a checkbox.

Try this: (1) Push requests per second to 5,000 at the defaults. Five targets stay healthy, each one takes 1,000 req/s, utilisation hits 500 percent, target serve time jumps to 190 ms, and the verdict flips to saturated. Same targets, same check. Only the traffic changed. (2) Now set requests back to 600 and uncheck "Health check catches the sick target". 16.7 percent of requests, 100 a second, hit the sick target and fail. The load on the healthy targets does not move. The dashboards stay green, because the check still passes. That is Lesson 06 again, and it is the failure this lesson is about. (3) Set sick targets to 0 and targets to 8 at 600 req/s. Each target takes 75 req/s, utilisation falls to 38 percent, and the verdict is comfortable. Then set sick to 2 with the check honest. The pool drops to 6, each takes 100 req/s, still comfortable, and zero requests fail. The check is what makes the pool honest.

Formulas, in plain terms. Healthy targets = targets minus sick. Effective pool = healthy targets when the check is honest, all targets when it lies. Load per target = requests per second divided by the effective pool. One target serves 200 req/s. Utilisation = load per target divided by 200. Sick share = sick divided by targets, only when the check lies, else zero. Requests failing = requests per second times the sick share. Target serve ms = 18 plus 40 for every point of utilisation past 70 percent when the check is honest, and 25 when the check lies, because the sick target takes 25 ms to fail. Moving a control updates the target serve time on hop 7, the budget bar, and the step panel above.

Lab: build the shop ALB, the billing ALB, and the NLB

Do this now, in your own terminal. You will write one Terraform file, prove it is valid without any cloud account, and prove SNI with two hostnames on one port, on your own machine. The apply step needs your own cloud account, and it says so plainly where it starts. Nothing before that step does. The config below is the worked setup from this lesson: ALB plus target group plus health check plus listener, then the same shape as an NLB.

Step 1: check the tools

terraform version
aws --version

Versions that ran for this lesson: Terraform 1.9.8. Any 1.5 or newer works. aws --version should print a version. If the AWS CLI is not installed, install AWS CLI v2. The apply step needs it. Nothing before the apply step does, and the fmt, init, and validate steps below all run with no cloud account and no credentials.

Step 2: write main.tf

Save this as main.tf in an empty folder. Every value you do not know yet is a variable at the top. Fill them in from your own account only when you reach the apply step.

terraform {
  required_version = ">= 1.5"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

# ---------- shared inputs: fill these in from your own account ----------
variable "vpc_id" {
  type        = string
  description = "VPC the load balancers live in"
}

variable "public_subnet_ids" {
  type        = list(string)
  description = "Two public subnets in different AZs, for the internet-facing ALB"
}

variable "private_subnet_ids" {
  type        = list(string)
  description = "Two private subnets in different AZs, for the internal ALB and the NLB"
}

variable "shop_cert_arn" {
  type        = string
  description = "ACM certificate ARN for shop.example.com (DNS validation, see Lesson 149)"
}

variable "api_cert_arn" {
  type        = string
  description = "ACM certificate ARN for api.example.com, the second SNI name on the same listener"
}

# ---------- the shop: internet-facing ALB ----------
resource "aws_lb" "shop" {
  name               = "shop-alb"
  internal           = false # internet-facing: it gets a public DNS name
  load_balancer_type = "application"
  subnets            = var.public_subnet_ids
}

resource "aws_lb_target_group" "shop" {
  name     = "shop-tg"
  port     = 8080
  protocol = "HTTP"
  vpc_id   = var.vpc_id

  # The health check asks the app to prove it can serve, not just that
  # the process answers. Lesson 06 showed what a shallow check costs:
  # a target whose database connection died kept passing a root-path
  # check, and one in six requests failed while the dashboards stayed green.
  health_check {
    path                = "/healthz"
    port                = "8080"
    protocol            = "HTTP"
    healthy_threshold   = 2
    unhealthy_threshold = 2
    interval            = 15
    timeout             = 5
  }

  # Connection draining: when a target leaves, in-flight requests get
  # this long to finish instead of being cut off mid-checkout.
  deregistration_delay = 30
}

resource "aws_lb_listener" "shop_https" {
  load_balancer_arn = aws_lb.shop.arn
  port              = 443
  protocol          = "HTTPS"
  ssl_policy        = "ELBSecurityPolicy-TLS13-1-2-2021-06"
  certificate_arn   = var.shop_cert_arn # the default certificate

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.shop.arn
  }
}

# SNI: the second hostname rides the same listener on the same port.
# The client names api.example.com in the first TLS message, and the
# listener serves this certificate instead of the default.
resource "aws_lb_listener_certificate" "shop_api" {
  listener_arn    = aws_lb_listener.shop_https.arn
  certificate_arn = var.api_cert_arn
}

# Host + path routing: /api/* on api.example.com goes to the API group.
resource "aws_lb_listener_rule" "shop_api_path" {
  listener_arn = aws_lb_listener.shop_https.arn
  priority     = 10

  action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.api.arn
  }

  condition {
    host_header {
      values = ["api.example.com"]
    }
  }
  condition {
    path_pattern {
      values = ["/api/*"]
    }
  }
}

resource "aws_lb_target_group" "api" {
  name     = "api-tg"
  port     = 8080
  protocol = "HTTP"
  vpc_id   = var.vpc_id

  health_check {
    path                = "/healthz"
    healthy_threshold   = 2
    unhealthy_threshold = 2
    interval            = 15
    timeout             = 5
  }
  deregistration_delay = 30
}

# ---------- the billing worker: internal ALB, same shape, no public face ----------
resource "aws_lb" "billing" {
  name               = "billing-alb"
  internal           = true # internal: private DNS name only, no public IP
  load_balancer_type = "application"
  subnets            = var.private_subnet_ids
}

resource "aws_lb_target_group" "billing" {
  name     = "billing-tg"
  port     = 8080
  protocol = "HTTP"
  vpc_id   = var.vpc_id

  health_check {
    path                = "/healthz"
    healthy_threshold   = 2
    unhealthy_threshold = 2
    interval            = 15
    timeout             = 5
  }
  deregistration_delay = 30
}

resource "aws_lb_listener" "billing_http" {
  load_balancer_arn = aws_lb.billing.arn
  port              = 80
  protocol          = "HTTP"

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.billing.arn
  }
}

# ---------- the same shape as an NLB: TCP, no HTTP rules, no TLS on the box ----------
resource "aws_lb" "shop_nlb" {
  name               = "shop-nlb"
  internal           = false
  load_balancer_type = "network"
  subnets            = var.public_subnet_ids

  # NLB cross-zone is off by default in some setups and on in others.
  # Say it out loud either way: off means each AZ serves only its own
  # targets, and one AZ with fewer targets gets more load per target.
  enable_cross_zone_load_balancing = true
}

resource "aws_lb_target_group" "shop_nlb" {
  name     = "shop-nlb-tg"
  port     = 443
  protocol = "TCP"
  vpc_id   = var.vpc_id

  health_check {
    protocol            = "TCP"
    healthy_threshold   = 2
    unhealthy_threshold = 2
    interval            = 15
    timeout             = 5
  }
  deregistration_delay = 30
}

resource "aws_lb_listener" "shop_nlb_tls" {
  load_balancer_arn = aws_lb.shop_nlb.arn
  port              = 443
  protocol          = "TLS"
  ssl_policy        = "ELBSecurityPolicy-TLS13-1-2-2021-06"
  certificate_arn   = var.shop_cert_arn

  default_action {
    type             = "forward"
    target_group_arn = aws_lb_target_group.shop_nlb.arn
  }
}

Step 3: format it, then prove it is valid

terraform fmt
terraform init -backend=false
terraform validate

Expected output, exactly as it ran in this build. fmt prints nothing when the file is already formatted. That silence is the pass. init downloads the AWS provider, and validate checks the config against it:

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

Success! The configuration is valid.

One honest note. The provider download is a few hundred MB. My first init failed with "no space left on device" because /tmp on that machine was a 512 MB tmpfs. If you hit that, point the temp dir at a bigger disk and run it again: TMPDIR=$HOME/.tmp terraform init -backend=false. The config was fine. The disk was full.

Step 4: prove SNI with two hostnames, on your own machine

Four words from Lesson 149 first, because this step uses them. A listener is the front door of a load balancer: a port, a protocol, and a certificate. ACM is AWS Certificate Manager. It asks a public CA for certificates, holds the private key, and renews the certificate before it ends. DNS validation is the proof: ACM gives you a record to put in DNS, and when the CA can read it back, the domain is yours. SNI is the name in the first TLS message. The client sends the hostname before the server picks a certificate, so one listener can serve many names. In the cloud, ACM holds the certificates and the listener picks by SNI. Here, two lab certificates and a small Python server play the listener's part, so you can watch SNI pick.

Make two lab certificates, one per name:

openssl req -x509 -newkey rsa:2048 -keyout app.key -out app.crt -days 90 -nodes -subj "/CN=app.local" -addext "subjectAltName=DNS:app.local"
openssl req -x509 -newkey rsa:2048 -keyout api.key -out api.crt -days 90 -nodes -subj "/CN=api.local" -addext "subjectAltName=DNS:api.local"

Then save this as sni_lab.py in the same folder, and run it with python3 sni_lab.py:

import http.server, ssl

CERTS = {"app.local": ("app.crt", "app.key"),
         "api.local": ("api.crt", "api.key")}

def make_context(certfile, keyfile):
    ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
    ctx.load_cert_chain(certfile, keyfile)
    return ctx

CONTEXTS = {name: make_context(*pair) for name, pair in CERTS.items()}

def pick(conn, hostname, _initial_ctx):
    # SNI: the client names the host in ClientHello, before any cert is sent.
    conn.context = CONTEXTS.get(hostname, CONTEXTS["app.local"])

base = CONTEXTS["app.local"]
base.sni_callback = pick
srv = http.server.HTTPServer(("127.0.0.1", 8443), http.server.SimpleHTTPRequestHandler)
srv.socket = base.wrap_socket(srv.socket, server_side=True)
srv.serve_forever()

Then ask for each name, in another shell. Expected output, exactly as it ran in this build:

echo | openssl s_client -connect 127.0.0.1:8443 -servername app.local 2>/dev/null | grep -E "subject=|Verify return code"
subject=CN = app.local
Verify return code: 18 (self-signed certificate)

echo | openssl s_client -connect 127.0.0.1:8443 -servername api.local 2>/dev/null | grep -E "subject=|Verify return code"
subject=CN = api.local
Verify return code: 18 (self-signed certificate)

echo | openssl s_client -connect 127.0.0.1:8443 2>/dev/null | grep -E "subject=|Verify return code"
subject=CN = app.local
Verify return code: 18 (self-signed certificate)

Same port, same server, different certificate per name. With no -servername the server falls back to the default, app.local. The return code on each of those is 18, self-signed, because these are lab certificates. That is expected here. The subject line is the proof that SNI picked the certificate. Stop the server with Ctrl-C when you are done.

Step 5: read the NLB back, and name the three differences

Open main.tf and read the NLB block against the ALB block. Three things change, and nothing else does. The type is network, not application. The target group health check is TCP, not HTTP, because the NLB never opens an HTTP request. And the listener is TLS, not HTTPS: it terminates TLS and forwards TCP. There are no listener rules, because there is no HTTP to route on. Same target group shape, same draining delay, same health check thresholds. The shape is the same. The layer is different.

Step 6: apply it in your own cloud account

This step needs your own cloud account, and I cannot run it here. There is no cloud account in this build. Everything below is for your account, and the outputs shown are the shape of what your own account returns, not output from this machine. Do not paste them into a report as if they ran here.

First the values. Save them as terraform.tfvars. Use your real IDs, not these placeholders:

vpc_id             = "vpc-0123456789abcdef0"
public_subnet_ids  = ["subnet-aaaa", "subnet-bbbb"]
private_subnet_ids = ["subnet-cccc", "subnet-dddd"]
shop_cert_arn      = "arn:aws:acm:us-east-1:123456789012:certificate/aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa"
api_cert_arn       = "arn:aws:acm:us-east-1:123456789012:certificate/bbbbbbbb-bbbb-bbbb-bbbb-bbbbbbbbbbbb"

The certificates come from ACM with DNS validation, the way Lesson 149 teaches it: request the certificate, put the validation record in DNS, wait for ISSUED. Then run:

terraform plan
terraform apply

Read the plan before you type yes. It should create three load balancers, four target groups, three listeners, one listener rule, and one extra listener certificate. Nothing else. Then prove the shop target group is healthy, from your account:

aws elbv2 describe-target-health --target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/shop-tg/0000000000000000

Expected output, the shape of it: a TargetHealthDescriptions list, one entry per target, each with State healthy and a reason. A target that says unhealthy here has failed the /healthz check twice in a row. Read the check, not the balancer. Then check the listener carries both certificates:

aws elbv2 describe-listener-certificates --listener-arn arn:aws:elasticloadbalancing:us-east-1:123456789012:listener/app/shop-alb/0000000000000000/0000000000000000

Expected output, the shape of it: a Certificates list with two entries, shop.example.com as the default and api.example.com beside it. That is SNI on the real listener, the same proof you ran locally in step 4. When you are done, tear it down: terraform destroy. A load balancer left running bills by the hour.

Check yourself

Change one thing and run it again. Set deregistration_delay to 5 in the shop target group and run terraform validate again. It still passes. Then run python3 lb_math.py from the Go deeper section and read how many requests still route to a leaving target at 5 seconds versus 30. The config change is one number. The difference is 5,000 requests. That is what the delay is.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

Cloud load balancing is not one thing. It is four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how many balancers you own. They are about how much of the traffic path is deliberate: the scheme, the type, the check, and the drain. Read them in order. Each level fixes the failure that hurt the most at the level below it.

Basic — One internet-facing ALB, default everything

At the basic level, the balancer is one internet-facing ALB with the defaults. One listener on 443, one target group, the health check on the root path, cross-zone on because that is the default, and a deregistration delay of 300 seconds because nobody changed it. It works. Traffic flows. The shop is up. That is the whole system, and it is how every cloud balancer starts.

Do this: build the shop ALB from the lab, with the defaults, and curl the shop through the balancer DNS name. Then register one target that returns 500 on every real request but 200 on /. Watch the balancer keep sending it traffic. That is the failure the next level fixes, and you have just seen it with your own eyes.

Uses: A demo for one stakeholder. A first version built to answer one question: does traffic flow through a balancer at all. I use this level to learn, never to serve real users.

Failure modes: The health check lies, because it only asks the root path. A deploy takes 300 seconds per target to drain, because the delay was never set. An internal service gets an internet-facing balancer by copy-paste, because the scheme was never a decision.

Intermediate — Internal where it must be private, and a check that proves serving

At the intermediate level, the scheme is a decision, not a default. Public services get internet-facing balancers. Private services get internal balancers, and the choice is written down next to the service name. The health check hits /healthz and runs a real database query. Two failures mark a target down, two successes bring it back. The deregistration delay is 30 seconds, chosen on purpose. This is the level this lesson teaches in full.

Do this now: run the lab in this lesson, then change the health check path back to / and deploy the sick target from the basic level again. Watch the balancer keep it in rotation. Then change the path back to /healthz and watch it leave. You just proved the check is the brains, not the balancer.

Uses: A small team serving a real shop and a real billing worker. An internal tool where a bad deploy is an annoyance, not an outage. Any system where the callers are people you can talk to when something breaks.

Failure modes: The check is deep but the alert is missing, so a target flaps all day and nobody notices. Cross-zone is off and one AZ has fewer targets, so that AZ's targets run hotter than the rest. The certificate renews fine but nobody checks the served cert, so a deleted validation record breaks traffic silently at renewal.

Advanced — Three balancer types, each for its own job

At the advanced level, the team owns all three types and picks by the traffic, not by habit. HTTP services get an ALB, with host and path rules doing the routing. TCP services and anything that needs a static IP gets an NLB. The firewall appliance gets a GWLB, and the appliance is a target like any other. The health check has an alert on per-target error rate, not just the pool total. Connection draining is a number the team chose from the traffic math, not a default they inherited.

Do this: take the lab and add a second target group behind the same ALB listener, with a host rule for a new hostname. Then add the NLB from the lab in front of a TCP service, and write down, in one sentence each, why the ALB fits the first and the NLB fits the second. If you cannot write the sentence, you have not picked. You have copied.

Uses: A production system where a bad balancer choice costs real users: a shop at peak, a billing worker that must stay private, a security appliance that must see every packet. A team that ships a new service every month and needs the balancer choice to be a habit, not a meeting.

Failure modes: The GWLB appliance becomes the bottleneck, because every packet passes through it and nobody scaled it. The NLB keeps a static IP per AZ and one AZ's IP gets allow-listed everywhere, so that AZ can never be drained. The ALB rules grow to fifty entries and nobody knows which rule catches a request anymore.

Master — The balancer as a platform

At the master level, the balancer is a platform, not a config file. Every service gets its balancer from a template: scheme, type, check, drain delay, and alerts, all in version control. A new service fills in five fields and gets the right balancer. The health check contract is written down: /healthz proves the database and the one downstream service the app needs, and it returns 503 otherwise. The served certificate is checked by a probe, not by the ACM status. At 10x the traffic, the ALB scales itself and the targets scale behind it. At 100x, the NLB takes the edge and the ALB sits behind it, terminating TLS where the routing decisions live.

Do this: write the five-field template for your team. Service name, public or private, HTTP or TCP, health check endpoint, drain delay in seconds. Then fill it in for three services you actually run, and check whether any of them has the wrong scheme or the wrong type today. Fix one. That fix is the platform working.

Uses: A platform team that serves other teams, where a wrong balancer is an outage for someone else. An org where the balancer choice is made once, in a template, and every service inherits it.

Failure modes: The template is right and the services drift, because somebody edited a balancer by hand and the template never ran again. The probe that checks the served certificate pages at 3 AM for a cert that renews at 4 AM. The GWLB appliance team and the platform team disagree about who owns the packet path, and the argument lasts longer than the outage.

Go deeper

ALB, NLB, GWLB: the one-line picker

ALB is Layer 7. It opens the HTTP request, reads the host and the path, and routes on them. Pick it when the traffic is HTTP or HTTPS and the routing decision needs the request itself. NLB is Layer 4. It routes TCP and UDP, and it can terminate TLS without ever reading HTTP. Pick it when the traffic is not HTTP, when you need a static IP per AZ, or when you need millions of requests a second with the lowest latency. GWLB is the appliance case. It forwards traffic to a third-party appliance, a firewall or an intrusion system, and the appliance decides what happens next. Pick it when something must sit in the path and inspect every packet. The three types do not compete. They answer different questions, and the question is always about the traffic, never about the brand.

What the NLB does that the ALB cannot

Three things. A static IP address per AZ, which matters when a partner allow-lists your address and will not take a DNS name. UDP, which the ALB does not speak. And TLS passthrough, where the NLB forwards the encrypted stream untouched and the target ends TLS itself. The ALB always ends TLS at the listener. That is a feature when you want the balancer to read the request, and a limit when the target must see the original TLS. The NLB also scales further and faster, because Layer 4 is cheaper than Layer 7. The price is that it never reads a host header, so it cannot route /api/* to one group and /checkout to another. That routing is the ALB's job.

Cross-zone: say it out loud

Cross-zone load balancing decides whether a request can land on a target in any AZ, or only in its own AZ. With it on, the balancer spreads evenly across all targets, and an AZ with two targets gets the same per-target load as an AZ with four. With it off, each AZ serves only its own targets. The AZ with fewer targets loads each one harder, and the AZ with more targets idles. The ALB has cross-zone on and it cannot be turned off. The NLB lets you choose, and the default has changed over time, which is why I say it out loud in the config instead of trusting the default. The lab sets it to true and says why. If your traffic must stay in its own AZ for cost or compliance, turn it off on purpose, and watch the per-AZ load, because the imbalance is the price.

Draining is a number, not a hope

When a target leaves the group, the deregistration delay is how long in-flight requests get to finish. The default is 300 seconds. At 1,200 requests a second across six targets, that is 60,000 requests still routed to a leaving target, and 360,000 across a rolling deploy of all six. Write the script below as lb_math.py and run it with python3 lb_math.py. It computes the draining numbers and the sick share from the lesson. No cloud account, no network. Plain arithmetic.

"""lb_math.py — the two numbers that decide a load balancer rollout.

1. Draining: when a target leaves the group, requests already routed to
   it keep arriving for the whole deregistration delay. In-flight = share
   of traffic x delay.
2. Sick share: when the health check lies, sick targets stay in rotation
   and take their fair share of requests. Those requests fail.

Run: python3 lb_math.py
No cloud account, no network. Plain arithmetic from the lesson's numbers.
"""


def drain_in_flight(rps, targets, delay_s):
    # Each target takes 1/targets of the traffic, for delay_s seconds.
    return rps / targets * delay_s


def sick_share(targets, sick, honest_check):
    if honest_check:
        return 0.0
    return sick / targets * 100


def per_target(rps, targets, sick, honest_check):
    pool = targets - sick if honest_check else targets
    return rps / pool


print("deregistration delay, 1200 req/s, 6 targets:")
for d in (5, 30, 300):
    print("  delay %3ds -> %7.0f requests still routed to the leaving target"
          % (d, drain_in_flight(1200, 6, d)))
print("  a 300 s delay on a deploy that rolls 6 targets, one at a time:")
print("  total drained through leaving targets:",
      round(sum(drain_in_flight(1200, 6, 300) for _ in range(6))))

print()
print("sick share, 6 targets, 1 sick:")
for honest in (True, False):
    print("  honest check: %-5s -> %4.1f%% of requests fail, %.0f req/s per healthy target"
          % (honest, sick_share(6, 1, honest), per_target(1200, 6, 1, honest)))
print("  same pool, 2 sick, check still lying: %.1f%% fail"
      % sick_share(6, 2, False))
print()
print("capacity check, 200 req/s per target, 1200 req/s arriving:")
for targets in (4, 6, 8):
    print("  %d targets, honest check -> %.0f req/s each, %.0f%% utilised"
          % (targets, per_target(1200, targets, 1, True),
             per_target(1200, targets, 1, True) / 200 * 100))

Expected output, exactly as it ran in this build:

deregistration delay, 1200 req/s, 6 targets:
  delay   5s ->    1000 requests still routed to the leaving target
  delay  30s ->    6000 requests still routed to the leaving target
  delay 300s ->   60000 requests still routed to the leaving target
  a 300 s delay on a deploy that rolls 6 targets, one at a time:
  total drained through leaving targets: 360000

sick share, 6 targets, 1 sick:
  honest check: True  ->  0.0% of requests fail, 240 req/s per healthy target
  honest check: False -> 16.7% of requests fail, 200 req/s per healthy target
  same pool, 2 sick, check still lying: 33.3% fail

capacity check, 200 req/s per target, 1200 req/s arriving:
  4 targets, honest check -> 400 req/s each, 200% utilised
  6 targets, honest check -> 240 req/s each, 120% utilised
  8 targets, honest check -> 171 req/s each, 86% utilised

Read the last three lines twice. At 1,200 requests a second, six targets are not enough even with an honest check. Each one takes 240 a second against a 200 capacity. Eight targets is the first size that fits. That is the capacity check doing its job, and it is cheaper than a page at noon.

TLS ends at the listener, and the Ingress is a different door

Lesson 44 taught TLS termination at the Ingress, inside Kubernetes. This lesson terminates TLS at the cloud load balancer, outside the cluster. Same cryptography, different door. The balancer ends TLS and forwards plain HTTP to the targets. The Ingress ends TLS and forwards plain HTTP to the pods. When both exist, TLS ends twice: once at the balancer, once at the Ingress. That is normal, and it is a choice, not an accident. The rule I use: terminate where the routing decision lives. If the ALB routes by host and path, it ends TLS. If the Ingress routes by host and path, it ends TLS. Terminate at the door that reads the request, and do not terminate at both doors unless you know why both doors need to read it.

What changes at 10x and 100x

At 10x the traffic, the ALB scales itself and the targets scale behind it. The health check gets stricter, not looser, because one bad target at that scale takes down a visible slice of users. The deregistration delay gets shorter, because a rolling deploy at 10x cannot wait 300 seconds per target. At 100x, the NLB takes the edge for the static IPs and the raw throughput, and the ALB sits behind it, terminating TLS where the routing decisions live. The GWLB appliance gets its own scaling plan, because every packet passes through it and it is the first thing to saturate. The target group does not change. The listener does not change. The parts around them change, and that is the point. The balancer is the front door. The building behind it is the system.

One time it broke

This one stayed with me because the failure was polite. Nothing crashed. Nothing paged. A third of the invoices simply did not go out, and finance found it before engineering did.

What broke

It was a Tuesday morning. The billing worker runs behind an internal ALB with three targets. A deploy went out to one target, and the deploy broke that target's database connection. The target process was fine. It answered every request the balancer sent it. The problem was that it could not do any real work, because every real request needed the database. The health check hit the root path and only asked if the process answered. The process answered. So the balancer kept sending one in three requests to a target that could not charge a single card. Invoices failed at random. Some customers were charged. Some were not. Finance noticed the gap in the morning numbers before any dashboard changed color.

Timeline

  • 09:12Deploy rolls to target 1 of 3. The database connection dies in the new code. The process stays up.
  • 09:14Health check asks /. Target 1 answers 200. It stays in rotation.
  • 09:20One in three invoice calls starts failing. The pool dashboard is green. The per-target error rate is not on any dashboard.
  • 10:05Finance reports the morning invoice count is a third short. Engineering is paged by a person, not by an alert.
  • 10:22Somebody reads the per-target errors. Target 1 has failed every real request since 09:12. The other two are clean.
  • 10:31Target 1 is deregistered by hand. Invoices recover. The deploy is rolled back.
  • Next dayThe team reads the incident back and finds the failure was in the check, not in the target. The target broke. The check lied.

What changed after

Three things, and I still do all three. First, the health check changed. It now hits /healthz, and /healthz runs a real database query and calls the one downstream service the worker needs. If any of those fail, the check returns 503. A process that is alive but cannot serve is not healthy. Second, I added an alert on error rate per target, not just on the pool total. A pool total hides one bad target behind two good ones. A per-target alert catches it in minutes. Third, I added a rule for deploys: after a deploy to one target, I watch that target's real error rate for ten minutes before the deploy rolls to the next one. A deploy that breaks one target quietly is a deploy I want to catch on the first target, not the third.

If you remember nothing else

Design exercise

Design the load balancing for a shop that peaks at 50,000 requests a second, a billing worker that charges cards and must stay private, and a firewall appliance that must inspect every packet before it reaches any server. The shop speaks HTTP. The billing worker speaks HTTP. The appliance is not a server.

Clarifying questions to ask

  • Who calls the shop, and from where? Is any of the traffic not HTTP, and does any partner need a static IP?
  • Who calls the billing worker, and is any caller outside the VPC? What happens if the billing worker is reachable from the internet for one hour?
  • What does the firewall appliance need to see: every packet, or only HTTP? What happens when the appliance is down?
  • How long can a checkout take, and what is the peak minute? How many targets does the shop need at 200 requests a second each?

Worked approach

Shape first. The shop gets an internet-facing ALB. The billing worker gets an internal ALB. The firewall appliance gets a GWLB, and the appliance is the target. Three balancers, three jobs, no sharing. The shop ALB carries two certificates through SNI, shop and api, on one listener. The billing ALB has one listener and one target group, and no public anything.

Then the numbers. 50,000 requests a second at 200 per target is 250 targets, and I run 300 so one AZ can lose targets without a page. The billing worker takes 400 a second across 3 targets, and the check is deep: /healthz runs a real database query. The deregistration delay is 30 seconds, so a leaving shop target still takes 30 seconds times its share: 50,000 divided by 300, times 30, is 5,000 requests. That is the number the deploy plan has to absorb.

Then the failures. A sick shop target leaves rotation when /healthz returns 503 twice. The per-target error rate has an alert, because the incident in this lesson taught me that a pool total hides one bad target. The GWLB appliance scales on its own numbers, and the plan says what happens when it is down: traffic stops, by design, because an appliance that must see every packet cannot be bypassed quietly.

Trade-offs to name. Three balancers cost more than one, and the GWLB adds a hop to every request. An internal billing worker cannot be reached for a quick manual check from a laptop, so debugging needs a jump host inside the VPC. The 300-target shop pool drains slowly on a full deploy, so deploys roll one AZ at a time. Every one of those costs is visible, and every one is smaller than the cost of a billing worker on the internet or a firewall the traffic walks around.

Checkable outcome

One page with a table: one row per service, naming the balancer type, the scheme, the listener, the target group, the health check endpoint, and the drain delay. Plus the draining number for the shop, worked out in the open, and one sentence per service saying why that balancer fits. If a row cannot name its health check endpoint, the design is not done.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series — Lesson 173
VPC Peering and Private Connectivity: Transit Gateway, PrivateLink, and the Road Home

Load balancers move traffic inside one VPC. The next lesson moves traffic between VPCs: peering, Transit Gateway, PrivateLink, and the road home.

Continue to Lesson 173 →