System Design · Lesson 102

Serverless

Lessons 99 to 101 gave you the clouds. This lesson gives you the model that runs on top of them. Serverless is code that runs when an event arrives, and stops when it is done. There is no server to keep running, no box to patch, no capacity to plan. One image upload shows the whole shape. The upload fires an event. A function wakes up, resizes the image, writes the thumbnail, and goes away. I learned serverless the way I learn most models in this job. I shipped a function that was fast alone, watched it throttle under a burst, and then read the limits until the limits confessed. This lesson walks one invocation through serverless, step by step, from a first function to the deep end: cold-start mitigation, state handling, cost at scale, and when not to use serverless at all.

Difficulty: Advanced · Know this first: Lesson 05, Lesson 26
Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one image-resize invocation, not a measurement from a live system, but the shape is honest: in most slow requests I have chased, one or two hops eat almost the whole budget. Step through the lesson or switch the scenario and watch the bar change. The cold start quintuples the trip without the handler doing anything different. Same code, same image. Only the runtime's temperature changed, and the bill quintupled.

Try it: change the load and watch the invocation bend

The budget bar above shows three fixed stories. This simulator lets you write your own. It models one image-resize invocation, with the traffic rate, the cold starts, the memory, and the downstream calls all in the mix. Move a slider and watch the handler hop, the cold-start cost, and the total change in the budget bar, in the hop timings, and in the status line. The numbers are a teaching example, not live measurements. The shape is the lesson: the handler is steady, the cold start is the variable, and the traffic rate is the multiplier.

How fast events arrive. Concurrent executions is invocations per second times the total time in seconds, so a faster rate and a slower invocation multiply each other.

Share of invocations that start cold. A cold start costs 420 ms, so the average cold-start cost is the share times 4.2 ms. At 5 percent that is 21 ms on every invocation.

Memory is also CPU in serverless. The handler hop is 23,040 divided by the memory in MB: 45 ms at 512 MB, 180 ms at 128 MB, about 8 ms at 3,008 MB.

Each downstream call costs 25 ms. Two calls are 50 ms. Five calls are 125 ms, and every one of them is a chance to wait on somebody else.

Total per invocation154 msfixed hops + handler + cold-start cost + downstream
Concurrent executions15invocations per second times total time in seconds
Handler hop45 ms23,040 divided by memory in MB
Cold-start cost21 msabout 14% of the total bill
Downstream hop50 ms25 ms times downstream calls
Fixed hops38 msevent source 3 + runtime pick 5 + output 20 + logs 4 + answer 6

Try this: (1) Leave the defaults: 100 per second, 5 percent cold, 512 MB, 2 downstream calls. The total is about 154 ms, with about 15 concurrent executions. (2) Push the cold share to 100 percent, the memory down to 128 MB, and the downstream calls to 5 at 1,000 per second. The total is about 763 ms, with about 763 concurrent executions. Same function, same code. Only the knobs changed. (3) Set everything back, then set the memory to 3,008 MB and watch the handler fall to about 8 ms. That is what memory buys, and the bill is memory times time, so the faster handler can cost less.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

Serverless looks like one big idea. It is not. It is four layers stacked on each other, and each layer only makes sense after the one below it. I learned it in this order, and I teach it in this order: get one function running on one event, then wire it to real services, then make it survive a burst and stay cheap, then run it at scale with the right boundaries. Skip a layer and the next one reads like magic. Build them in order and nothing in this lesson is magic.

Basic — One function, one event, one job

Serverless at the basic level is a function and an event. A function is a small piece of code with a handler, one job, one input, one output. An event is a fact that something happened: an upload, a message, a schedule tick. The event source mapping wires the event to the function. When the event fires, the platform runs the function, and when the function returns, the platform stops it. There is no server to keep running.

Types:

  • Functions: Lambda, Cloud Functions, Azure Functions, the same idea on three clouds.
  • Event sources: object storage, queues, HTTP, schedules.
  • The handler, the one function that does the job.
  • The runtime, the language environment the handler runs in.
  • The log, the receipt the platform emits for every run.

Uses: when a team has a small job that happens sometimes. A thumbnail on upload. A welcome email on signup. A nightly cleanup on a schedule. The first win is that a job that runs once an hour stops needing a server that runs all month.

Dimensions: Scale: holds up to a few thousand invocations a day without special care. Failure modes: the function is cold most of the time, so the first call after a quiet spell is slow; a function that keeps state between calls is a bug, because the next call may run on a different runtime. Cost: per invocation and per millisecond of run time; a function that runs once an hour costs almost nothing. Trade-offs: you pay the cold start before you get any of the scale, so a job that needs a warm answer every time is paying a wait it does not want.

Intermediate — Wiring functions to real services

The intermediate layer is the function wired to storage, a database, and a queue. The function reads from object storage, writes to a table, and sends a message to a queue for the next step. The wiring is IAM: the function role may read one bucket, write one table, and send to one queue, and nothing else. The function stays stateless, Lesson 05, and the state lives in the services, not in the function. The queue between two functions is the boundary, Lesson 26: each function is a service with a size of one, and the queue is the contract between them.

Types:

  • Storage triggers, an upload that fires a function.
  • Queue triggers, a message that fires a function.
  • HTTP triggers, a request that fires a function.
  • IAM roles for functions, least privilege per function.
  • Environment variables and secrets, the config the function reads at start.
  • Dead-letter queues, where failed events go instead of being lost.

Uses: when the job has more than one step. An upload triggers a resize, the resize triggers a thumbnail, the thumbnail triggers a notification. Each step is a function, and the queue between them is the handoff. A workflow stops being one big process and starts being a chain of small functions, and a failure stops being a crash and starts being a message in a dead-letter queue.

Dimensions: Scale: holds to hundreds of thousands of invocations a day, but the wiring grows with the workflow, because every step needs a role and every role needs a review. Failure modes: a function that retries a failed write three times is three writes, and a queue that redelivers a message is a second run of the same job, so the handler must be idempotent or the same work happens twice; a role that is too wide can touch buckets it was never meant to see. Cost: per invocation, per millisecond, and per downstream call, so a workflow of five functions costs five invocations per upload. Trade-offs: small functions buy a small blast radius and cost a chain that is harder to trace than one process; a queue buys a handoff and costs a second place where messages can wait.

Advanced — Surviving bursts, cold starts, and the bill

The advanced layer is the limits, the cold-start mitigation, and the cost control. The concurrency limit caps how many copies of a function may run at once. Provisioned concurrency keeps a few runtimes warm, so the first call after a quiet spell is fast. The memory setting is also the CPU setting, so more memory is a faster handler. The bill gets a budget and an alarm, because serverless charges per invocation and per millisecond, and a burst is a million invocations in an hour. The database gets a connection pooler, because a hundred functions each opening a connection is a hundred connections.

Types:

  • Concurrency limits, per function and per account.
  • Provisioned concurrency, warm runtimes kept ready.
  • Memory and timeout settings, the two knobs that shape every run.
  • Reserved concurrency, a cap that protects the database from a burst.
  • Connection poolers and data APIs, the way functions talk to databases without a connection each.
  • Cost tools: budgets, alarms, and per-function cost reports.

Uses: when the function has real traffic and a real on-call rotation. A flash sale that bursts to a thousand invocations a second. A thumbnail function that must answer fast even after a quiet night. A finance team that wants the bill explained before it arrives. The function becomes something you operate, not something you deployed once and hoped about.

Dimensions: Scale: the downstream services become the bottleneck; the function scales to thousands of copies, and the database allows a hundred connections; the limit that matters is the smallest one, and it is usually the database. Failure modes: a burst that hits the concurrency limit queues the invocations, and the queue looks like a slow function from the outside; a provisioned concurrency setting that is too low is money spent on warm runtimes that still go cold; a retry policy with no limit turns one failed write into five failed writes. Cost: operational; somebody has to own the bill, review the roles, and test the burst. Trade-offs: provisioned concurrency buys a warm answer and costs money whether the runtimes run or not; reserved concurrency buys a protected database and costs a cap that throttles the function under a burst; a bigger memory buys a faster handler and costs more per millisecond, so the faster handler can cost less or more, and the only way to know is to measure.

Master — When not to use serverless, and the platform on top

The master layer is the judgment about where serverless belongs, and the platform that runs it at scale. Not every job is a function. Long jobs, stateful jobs, and jobs that hold a connection belong on a server or a container. The platform team owns the shared pieces: the function templates, the IAM patterns, the CI pipeline that deploys the functions, and the cost reports. The app teams own their functions, and they build on the platform instead of rebuilding it. The design at this layer is a choice, made on purpose, with the trade-offs written down.

Types:

  • Step Functions and workflow orchestrators, the way long chains are run without one big function.
  • Container-based functions, a function packaged as a container for bigger jobs.
  • Event buses, the shared event system that many functions listen to.
  • Function templates and shared libraries, the platform's starting point for a new function.
  • Cost allocation by function and by team, the bill split so each team sees its own spend.
  • The exit plan: the function code kept portable, so a function that outgrows serverless can move to a container without a rewrite.

Uses: when there are many functions and the question is no longer how to write one, but where each job belongs. A short, stateless, event-driven job is a function. A long, stateful job is a container. A job that runs all day is a server. The platform team is the team that makes the other teams fast without making them unsafe, and the exit plan is the part that keeps the platform honest.

Dimensions: Scale: holds at thousands of functions and millions of invocations a day, because the cost moves from per-function heroics to shared patterns, and the config moves from hand-built functions to templates. Failure modes: a workflow orchestrator that chains twenty functions is twenty places to fail, and the failure is a state machine, not a log line; a shared event bus with no schema is a bus where every publisher can break every subscriber; a function that outgrows serverless and has no exit plan is a rewrite, not a move. Cost: a platform team; somebody has to run the templates, answer the tickets, and keep the guardrails current. Trade-offs: serverless buys no servers to run and costs a cold start, a time limit, and a bill that follows the invocation count; containers buy a long life and cost a server to run. The choice is not which is better. The choice is which fits the job, and the job is the thing that decides.

Go deeper

Cold-start mitigation

A cold start is a tax, and there are three ways to pay less of it. The first is a smaller package: a 5 MB deployment starts faster than a 250 MB one, so I keep the heavy libraries out and I load them only when the handler needs them. The second is a faster language: Go and Rust start faster than Java and .NET, because there is less runtime to build. The third is provisioned concurrency: a few runtimes kept warm, so the first call after a quiet spell is fast. The third one costs money whether the runtimes run or not, so I use it only for the functions where the wait matters. A thumbnail can wait. A login cannot.

State handling

A function has no memory between calls. That is the rule, and the rule is the design. If the handler needs state, the state lives in a service: a table, a cache, or a queue. The handler reads the state at the start, does the job, and writes the state at the end. The trap is the connection: a handler that opens a database connection and keeps it is holding a connection on a runtime that may never run again. The fix is a pooler, or a data API, or a connection opened and closed inside the handler. The handler is a guest in the runtime. It does not own the room.

Cost at scale

The serverless bill has three lines. Invocations, per million. Duration, per millisecond times memory. And downstream calls, per request. At a thousand invocations a day, the bill is pennies. At a million a day, the bill is real money, and the line that grows fastest is usually the duration, because a handler that runs for 500 ms costs ten times a handler that runs for 50 ms. I check the cost per invocation weekly, and I alarm on a daily spend that doubles, because a bill that doubles in a day is either a launch or a leak, and I want to know which one before the month ends. And I remember the free tier: the first million invocations a month are often free, which is why serverless feels free until it does not.

When not to use serverless

Serverless is wrong for four kinds of work. Long jobs: a function has a time limit, usually 15 minutes at most, and a job that takes 20 minutes cannot finish. Stateful jobs: a function has no memory between calls, and a job that needs to remember where it was needs a server. Long-lived connections: a function cannot hold a WebSocket open, and a job that holds one needs a process that lives as long as the connection. And constant traffic: a function that runs all day, every second, costs more than a server that runs all day, because the server is a flat rate and the function is a meter. The test is simple: if the job is short, stateless, and event-driven, it is a function. If it is long, stateful, or always on, it is not.

What changes at 10x and 100x

At 10x the traffic, the database becomes the bottleneck. The connection count runs out first, so I put a pooler in front of it, and I add a queue between the function and the database, so a burst becomes a line instead of a flood. At 100x, the design changes. The single function becomes a workflow, with a step function orchestrating the chain, and the chain gets a dead-letter queue at every step. The bill gets a budget line of its own, because at that scale the invocation count is a real slice of the spend, and a bill nobody budgets for is a bill nobody controls. And the cold start becomes a budget line too: at 100x, the cold starts are a real slice of p99, and a p99 nobody budgets for is a p99 nobody controls.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Thumbnail function — Lesson 102

Resources:
  UploadBucket:
    Type: AWS::S3::Bucket
  ThumbnailBucket:
    Type: AWS::S3::Bucket

  ThumbnailFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: src/
      Handler: app.lambda_handler
      Runtime: python3.12
      MemorySize: 512
      Timeout: 10
      ReservedConcurrentExecutions: 50
      ProvisionedConcurrencyConfig:
        ProvisionedConcurrentExecutions: 5
      Events:
        Upload:
          Type: S3
          Properties:
            Bucket: !Ref UploadBucket
            Events: s3:ObjectCreated:*
      Policies:
        - S3ReadPolicy:
            BucketName: !Ref UploadBucket
        - S3WritePolicy:
            BucketName: !Ref ThumbnailBucket

One function, one story. The function runs Python, with 512 MB of memory, so it gets a fair slice of CPU, and a 10 second timeout, so a stuck run is cut off instead of hanging. Reserved concurrency caps it at 50 concurrent runs, so a burst cannot flood the database with connections. Provisioned concurrency keeps 5 runtimes warm, so the first call after a quiet spell is fast. The event is an S3 upload, so the function runs when a photo arrives and never otherwise. The policies allow exactly two things: read the upload bucket, write the thumbnail bucket. Read the template together and you can see the whole trip: trigger, limits, warm pool, least privilege. Change one of them without reading the other three and you have changed the trip in a way you did not intend.

One time it broke

This one stayed with me because every dashboard stayed green while the users waited. The functions were fast, the database was full, and the burst did exactly what bursts do.

What broke

A photo app ran its thumbnail job on serverless functions. A flash sale drove a burst of uploads: a thousand a second, where the normal rate was ten. The functions scaled to a thousand copies in seconds, because that is what serverless does. Each copy opened a database connection to record the thumbnail. The database allowed a hundred connections. The 101st function got a refusal, retried, and got a refusal again. The retries piled up, the queue filled, and the thumbnails stopped. The function dashboards stayed green, because the functions were running and failing fast, and the database dashboard stayed green, because the database was answering, just refusing. The users saw uploads that never got thumbnails, and the on-call saw a system where every part was healthy and the whole was broken.

Timeline

  • 18:00Flash sale starts; uploads jump from 10 a second to 1,000 a second.
  • 18:01Functions scale to 1,000 concurrent copies; each opens a database connection.
  • 18:02Database hits its 100-connection limit; new functions get connection refusals and retry.
  • 18:05Retries pile up; the queue fills, and thumbnails stop appearing; users report uploads without thumbnails.
  • 18:12On-call reads the function logs; the logs show connection refusals, not handler errors; the database is the limit.
  • 18:20Reserved concurrency is set to 50, so the functions cannot open more than 50 connections; a queue is put between the function and the database.
  • 18:35Thumbnails resume; the backlog drains over the next hour, and the sale ends with every upload processed.

What changed after

Three things. First, every function that talks to a database now has reserved concurrency set, so the function copies cannot outnumber the database connections. The cap is the contract between the function and the database. Second, a queue sits between the function and the database, so a burst becomes a line instead of a flood, and the line drains at the speed the database allows. Third, the handler was made idempotent, so a retry is safe: running the same thumbnail job twice writes the same thumbnail once. A function that retries is a function that runs twice, and twice has to be safe. The functions did what they were told in this incident. The lesson is that what they were told was wrong, and the limit was the bug.

If you remember nothing else

Design exercise

The prompt

Design the serverless setup for an image-sharing app with 200,000 users. Uploads trigger a resize function, the resize triggers a thumbnail function, and the thumbnail triggers a notification. The app must survive a burst of 1,000 uploads a second. The database allows 100 connections. The bill must stay under $500 a month at the expected load. Sketch the functions, the queues, the concurrency limits, and the cost controls, and say what you would measure.

Clarifying questions to ask first

  • How many uploads a day, and how big is the average image, because the function bill follows the invocations and the milliseconds, not the users?
  • How fresh must the thumbnail be, and what happens when the function is cold: does the user wait, or see the image without a thumbnail first?
  • Is the $500 budget for functions only, or for functions plus the database and the storage, and which part of the bill is allowed to grow first?
  • Who owns the concurrency limits: one platform team, or does each team set its own, and who reviews a limit change?
  • What happens when the database is slow: does the function retry the write, or send the event to a dead-letter queue, and where does that retry live?

Rough sizing, worked with Little's Law

Start with the budget from this lesson. One invocation takes about 154 ms end to end at the simulator defaults: fixed hops about 38 ms, handler about 45 ms, cold-start cost about 21 ms, and downstream about 50 ms. At 200,000 users making one upload a day, that is about 2.3 uploads a second on average, and a burst of 1,000 a second in the evening. Little's Law for the sizing: at 1,000 uploads a second, each spending 0.154 seconds in the system, the number inside the system at once is 1,000 times 0.154, which is 154 concurrent executions, so the function needs a concurrency limit above 154, but the database allows only 100 connections, so the limit that matters is the database, not the function. I set reserved concurrency to 50, put a queue between the function and the database, and let the queue drain at 50 a second. The burst of 1,000 a second becomes a line of 50 a second, and the line drains in 20 seconds. The bill follows the invocations: 200,000 uploads a day is 6 million a month, three functions per upload is 18 million invocations, and at about $0.20 per million that is about $4 for invocations, plus duration, which is where the real money is. I alert at 80 percent of the budget, and I check the cost per invocation weekly.

Trade-offs to name

Provisioned concurrency buys a warm answer and costs money whether the runtimes run or not. Reserved concurrency buys a protected database and costs a cap that throttles the function under a burst. A queue buys a handoff and costs a second place where messages can wait. More memory buys a faster handler and costs more per millisecond, so the faster handler can cost less or more, and the only way to know is to measure. And the target is the database, not just the latency. A design that answers in 100 ms and floods the database with 1,000 connections fails this exercise, and the reserved concurrency cap is where the safety is.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series
Lesson 103 — Edge Computing
Coming next