Platform Engineering · Lesson 218

AI on AWS: Bedrock, SageMaker, and the Bill for Thinking

Your team asks the same questions every day. How do I reset the VPN. Where is the deploy runbook. Why did the bill jump. A support assistant that answers from your own runbooks would save hours a week, and AWS has the parts to build it. Bedrock rents you the model. A Knowledge Base rents you the memory. SageMaker rents you the training and the small models around it. This lesson wires them together into one request, then reads the bill that request creates. The model is the slow hop. It is also the dear hop. That is the whole lesson in one line.

Difficulty: Advanced · Know this first: Lesson 111, Lesson 112, Lesson 114, Lesson 16, Lesson 87, Lesson 177, Lesson 215, Lesson 181, Lesson 35

What you will build

By the end of this lesson you will have priced one AI request on AWS, line by line, and you will have the parts to run it. First, a token-cost calculator that turns prompt size and user count into a monthly bill. Second, a guardrail filter that blocks a bad prompt before it spends a token. Third, a small retrieval engine that finds the right runbook chunk for a question, the way a Knowledge Base does. Fourth, a Step Functions pipeline that runs the whole trip in order: guardrail, retrieve, rerank, generate, log. You will also read the live state of the services in your own account with the AWS CLI.

What you need: your own AWS account and the AWS CLI for the read-only checks, and Python 3 for the local parts. Nothing in this lesson runs against a live cloud from this page. The Python parts run on this machine and their outputs are shown exactly as they ran. The cloud parts run in your own account, and their outputs are marked that way. Your account IDs and resource names will differ.

The story: one question, nine hops, one bill

An engineer types a question into the team chat. How do I reset the VPN client. The question travels through nine hops and comes back as an answer. Every hop is an AWS service with a job and a price. The table below is the whole design on one page. Read it top to bottom, then walk the diagram hop by hop.

ServiceIts job in this designThe check you run
Bedrock InvokeModelRents the model by the token. Claude 3.5 Sonnet writes the answer.Read InputTokenCount and OutputTokenCount in CloudWatch.
Bedrock Knowledge BasesSearches the runbooks in S3 and hands the best chunks to the model.Ask a question only a runbook can answer.
Bedrock AgentsRuns the multi-step jobs: look up the ticket, then check the deploy, then answer.Count the agent steps per question. More than five is a loop.
Bedrock GuardrailsReads the question before the model does. Blocks denied topics, words, and PII.Read the blocked count. Zero blocks in a month means it is not wired up.
SageMaker trainingTrains the small reranker on your own questions and answers.Read the training job status. Failed means the data, not the model.
SageMaker endpointsServes the reranker all day on ml.c5.xlarge, $0.204 an hour.Read EndpointStatus. InService is the only good answer.
SageMaker JumpStartStarts from a ready model instead of training from zero.Compare its accuracy to your trained model before you pick one.
GPU instances and EKSRuns your own serving engine, like vLLM from Lesson 181, on your own nodes.Read GPU use on the node. A GPU at 5 percent is a bill, not a server.
S3Holds the runbooks, the training data, and the log of every answer.List the bucket prefix. A missing log is a hole in the audit trail.
CloudWatchCounts the tokens, the latency, the blocks, and the throttles.Open the dashboard. One graph per hop metric.
IAM, KMS, VPCDecide who can call the model, who can read the key, and which network the call travels on.Read the role policy. One model, one bucket, one key.
Lambda and Step FunctionsThe router and the pipeline. Lambda decides the route. Step Functions runs the steps in order.Read the Step Functions execution history. A failed step names itself.
Secrets Manager and Parameter StoreHold the helpdesk token and the config. The code holds neither.Read the secret with the CLI. The value never appears in a log.

The fork in this design is Bedrock or SageMaker. Bedrock is the rented model. You call it, you pay per token, you never see a server. SageMaker is the owned model. You train it, you serve it on an endpoint, you pay by the hour whether it serves or not. The rule I use is simple. If a rented model answers well enough, rent it. If the job is small and specific, like reranking five chunks, a small owned model on a cheap endpoint beats paying the big model to do the same sorting. This design does both. Bedrock writes the answer. SageMaker sorts the evidence.

The other fork is an Agent or a Lambda router. A Bedrock Agent decides its own steps. It can look up a ticket, then check a deploy, then answer, all in one call. That freedom is the risk. An agent that loops spends tokens on every lap, and the bill is the only alarm that fires. A Lambda router decides the steps in code. It is dumber and it is cheaper, and when it breaks you can read the exact line that broke. Start with the router. Move to the agent when the job has steps you cannot list in advance. Go deeper works both forks with numbers.

The price to remember is $3.00 in and $15.00 out per million tokens for Claude 3.5 Sonnet on demand in us-east-1. Output costs five times input. Every word the model writes costs five times a word it reads. That is why the prompt stays short and the answer stays shorter. The simulator below turns those two prices into a monthly bill.

Do this now, in your own account

Open the Bedrock console. Open the model list and find Claude 3.5 Sonnet. Write down its input and output price per million tokens: $3.00 and $15.00. Then open SageMaker and list your endpoints. Write down each endpoint name and its instance type. If you have no endpoint, write down zero. An endpoint you forgot is the most expensive line in this lesson, because it bills by the hour while you are not looking.

Run it as:

When it breaks, I check

One thing to remember

Four paths, one family

The diagram above follows one question through the AI design. These four paths are the same idea drawn four ways. One question, answered. One question, blocked. One dollar trail, hop by hop. One afternoon at peak, mostly waiting. Run each path. One run takes 7.2 seconds across the path, so you can watch each hop light up in turn.

The return rule: the green dot only starts moving after the orange dot reaches the data hop. A return with no request is a lie. The diagram never draws one. The failure replay at the end of Run all four shows the other side of the rule. Run it and watch the orange dot stop at the guardrail and never come back.

Path 1: the answered question

Cost trail: skipped

One question walks the design from the story. API Gateway, Lambda router, Guardrails, Knowledge Base, Bedrock model. The model writes the answer in 380 ms, and the answer starts back. The rerank and the logging happen around it, and they are in the main diagram.

Client0 ms
API Gateway4 ms
Lambda6 ms
Guardrails8 ms
Knowledge Base45 ms
Bedrock model380 ms

Not run yet.

Path 2: the blocked prompt

Knowledge Base: skipped

Same question, with a denied word pasted in from an error message. The guardrail reads it before the model does and blocks it. The Knowledge Base never sees the question. The model spends zero tokens. The user gets a short refusal instead of an answer.

Client0 ms
API Gateway4 ms
Lambda6 ms
Guardrailsblocked
Knowledge Baseskipped

Not run yet.

Path 3: the dollar trail

Guardrails: skipped

The same design, priced hop by hop at the simulator defaults: 10 users, 60,000 requests a month, 800 tokens in and 300 out per request. The trail adds up to $575.17 a month. The model is the big line. The endpoint is the fixed line.

Client$0.00
API Gateway$0.21
Lambda$0.04
Bedrock model$414.00
SageMaker rerank$148.92
S3 + CloudWatch$12.00

Not run yet.

Path 4: the throttled afternoon

Rerank: skipped

The same question at peak, with 50 users asking at once. Each hop shows its time. The model hop waits in a queue and then retries, and it costs 1,400 ms instead of 380 ms. The rest of the hops do not change. Throttling is a model problem, and the diagram shows it in one hop.

Client0 ms
API Gateway4 ms
Lambda6 ms
Guardrails8 ms
Knowledge Base45 ms
Bedrock modelqueued 1,400 ms

Not run yet.

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one question to the support assistant, not a measurement from a live system, but the shape is honest. The model hop is the big one. It costs 380 ms at the simulator defaults, and it costs $414.00 a month in tokens. The other hops are plumbing. Switch the scenario and watch the bar change. In the blocked story the model hop disappears, because a blocked prompt never reaches the model. In the throttled story the model hop grows to 1,400 ms, because the request waits in a queue and then retries. Move a simulator slider and watch the Knowledge Base hop and the model hop move with it.

Tokens or users: move the sliders and watch the bill

The budget bar above shows three fixed stories. This simulator lets you write your own. It answers the trade-off at the heart of this lesson: the model charges per token, the endpoint charges per hour, and the users multiply both. You set how many tokens go in, how many come out, how many runbook chunks the Knowledge Base fetches, and how many users ask at once. Move a slider and watch the bill, the model hop, and the budget total change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: output tokens cost five times input tokens, and the endpoint bills whether it serves or not.

800 tokens

The question plus the retrieved runbook chunks. More chunks means a longer prompt, and a longer prompt costs more per request.

300 tokens

The answer the model writes. Output tokens cost $15.00 per million, five times the $3.00 input price. Short answers are a cost control.

5 chunks

How many chunks the Knowledge Base hands to the model. Zero chunks means the model answers from memory, and the answer is a guess.

10 users

How many people ask at once. Each user makes about 200 requests a day. Past 20 users the model starts queueing, and the queue wait lands on the model hop.

Try this: (1) At the defaults, 800 tokens in, 300 out, 5 chunks, 10 users, the model hop is 380 ms and the Knowledge Base hop is 45 ms. The bill is $575.17 a month: $414.00 of Bedrock tokens, $148.92 of SageMaker endpoint, $12.00 of S3 and CloudWatch, $0.25 of API Gateway and Lambda. The budget total is 525 ms. (2) Drop to 200 tokens in, 100 out, 2 chunks, 5 users. The model hop falls to 130 ms and the Knowledge Base hop to 27 ms. The bill falls to $224.06 a month. The endpoint still costs $148.92, because it bills by the hour whether it serves or not. The budget total is 257 ms. (3) Push to 4,000 tokens in, 2,000 out, 10 chunks, 50 users. The model hop grows to 2,480 ms, with 450 ms of queue wait inside it. The bill is $12,762.01 a month, and $12,600.00 of it is Bedrock tokens. The budget total is 2,655 ms. From the defaults, the big prompt costs $12,186.84 more a month and waits 2,100 ms longer per request. That is the trade, priced.

Formulas, in plain terms. Requests per month = users times 6,000. Knowledge Base hop = 15 ms plus 6 ms per chunk. Queue wait = 0 below 20 users, then 15 ms per user past 20. Model hop = 30 ms plus 0.25 ms per input token plus 0.5 ms per output token plus the queue wait. Bedrock bill = requests times input tokens times $3.00 plus output tokens times $15.00, all over one million. SageMaker endpoint bill = $0.204 an hour times 730 hours, which is $148.92 a month. S3 plus CloudWatch is $12.00 a month. API Gateway plus Lambda is requests times $3.50 per million plus $0.04. Prices are the teaching prices from AWS published pricing at the time of writing, us-east-1 on demand. Your model and your region will differ. The shape will not.

Lab: run the AI design yourself

Do this now, in your own account and on your own machine. You will price the design, block a bad prompt, retrieve a runbook chunk, validate the pipeline, read the live state of the services, and train the reranker. The Python parts run on this machine and their outputs are shown exactly as they ran. The cloud parts run in your own account, and their outputs are marked that way. Your IDs will differ. The shapes will not.

Step 0: price the design on your own machine

Save this file as bedrock_cost.py. It takes four numbers, input tokens, output tokens, chunks, and users, and prints the bill the simulator shows. The defaults are the simulator defaults. Run it three times and check the totals against the simulator.

"""Token-cost calculator for the Lesson 218 AI-on-AWS design.

Usage: python3 bedrock_cost.py [in_tokens out_tokens chunks users]
Defaults are the simulator defaults: 800 in, 300 out, 5 chunks, 10 users.

Prices (us-east-1, teaching prices from AWS published pricing, Nov 2026):
  Bedrock Claude 3.5 Sonnet on-demand: $3.00 / 1M input, $15.00 / 1M output.
  SageMaker real-time endpoint ml.c5.xlarge: $0.204/hr x 730 hr = $148.92/mo.
  S3 + CloudWatch: $12.00/mo flat teaching figure.
  API Gateway REST: $3.50 per million requests. Lambda: $0.04/mo teaching figure.
Requests per month = users x 6000 (200 requests per user per day).
"""
import sys

IN_PRICE = 3.00
OUT_PRICE = 15.00


def main():
    args = [int(a) for a in sys.argv[1:]]
    in_tok, out_tok, chunks, users = (args + [800, 300, 5, 10])[:4]
    kb_ms = 15 + 6 * chunks
    queue = max(0, (users - 20) * 15)
    model_ms = 30 + 0.25 * in_tok + 0.5 * out_tok + queue
    reqs = users * 6000
    bedrock = reqs * (in_tok * IN_PRICE + out_tok * OUT_PRICE) / 1_000_000
    sagemaker = 0.204 * 730
    storage = 12.00
    api_lambda = reqs * 3.50 / 1_000_000 + 0.04
    total = bedrock + sagemaker + storage + api_lambda
    budget = 0 + 4 + 6 + 8 + kb_ms + 60 + model_ms + 12 + 10
    print(f"requests per month: {reqs}")
    print(f"knowledge base hop: {kb_ms} ms")
    print(f"model hop: {model_ms:.0f} ms (queue wait {queue} ms)")
    print(f"Bedrock tokens: ${bedrock:.2f} / mo")
    print(f"SageMaker endpoint: ${sagemaker:.2f} / mo")
    print(f"S3 + CloudWatch: ${storage:.2f} / mo")
    print(f"API Gateway + Lambda: ${api_lambda:.2f} / mo")
    print(f"total bill: ${total:.2f} / mo")
    print(f"budget total: {budget:.0f} ms")


if __name__ == "__main__":
    main()
python3 bedrock_cost.py

Expected output, exactly as it ran on this machine:

requests per month: 60000
knowledge base hop: 45 ms
model hop: 380 ms (queue wait 0 ms)
Bedrock tokens: $414.00 / mo
SageMaker endpoint: $148.92 / mo
S3 + CloudWatch: $12.00 / mo
API Gateway + Lambda: $0.25 / mo
total bill: $575.17 / mo
budget total: 525 ms
python3 bedrock_cost.py 200 100 2 5

Expected output, exactly as it ran on this machine:

requests per month: 30000
knowledge base hop: 27 ms
model hop: 130 ms (queue wait 0 ms)
Bedrock tokens: $63.00 / mo
SageMaker endpoint: $148.92 / mo
S3 + CloudWatch: $12.00 / mo
API Gateway + Lambda: $0.14 / mo
total bill: $224.06 / mo
budget total: 257 ms
python3 bedrock_cost.py 4000 2000 10 50

Expected output, exactly as it ran on this machine:

requests per month: 300000
knowledge base hop: 75 ms
model hop: 2480 ms (queue wait 450 ms)
Bedrock tokens: $12600.00 / mo
SageMaker endpoint: $148.92 / mo
S3 + CloudWatch: $12.00 / mo
API Gateway + Lambda: $1.09 / mo
total bill: $12762.01 / mo
budget total: 2655 ms

Three runs, three totals, and all three match the simulator. The small run shows the lesson of this design in one line. The tokens fell from $414.00 to $63.00, and the endpoint still costs $148.92. The endpoint is the floor. You cannot prompt your way below it. You can only delete it.

Step 1: block a bad prompt before it spends a token

Save this file as guardrail.py. It is the guardrail in miniature. A denied word or a denied topic blocks the prompt, and a blocked prompt spends zero tokens. On Bedrock the same decision is made by the Guardrails policy attached to the model call.

"""Guardrail filter, the way Bedrock Guardrails works in miniature.

Usage: python3 guardrail.py "your prompt here"
A prompt is BLOCKED when it contains a denied word or asks for a secret.
Everything else is ALLOWED and passes to the model. The denied list here
is the teaching stand-in for a Guardrails denied-topics policy and a
word filter. On Bedrock the same decision is made by the Guardrails
policy attached to the model call, before any token is generated.
"""
import sys

DENIED_WORDS = ["password", "api key", "secret key", "ssn"]
DENIED_TOPICS = ["how to pick a lock"]


def check(prompt):
    lowered = prompt.lower()
    for word in DENIED_WORDS:
        if word in lowered:
            return "BLOCKED", f"denied word: {word}"
    for topic in DENIED_TOPICS:
        if topic in lowered:
            return "BLOCKED", f"denied topic: {topic}"
    return "ALLOWED", "no policy hit"


def main():
    prompt = " ".join(sys.argv[1:]) or "How do I reset the VPN client?"
    verdict, reason = check(prompt)
    print(f"prompt: {prompt}")
    print(f"verdict: {verdict}")
    print(f"reason: {reason}")
    print(f"tokens spent: {0 if verdict == 'BLOCKED' else 'prompt tokens only'}")


if __name__ == "__main__":
    main()

Run it with a prompt that asks for a secret:

python3 guardrail.py "What is my api key for the deploy account?"

Expected output, exactly as it ran on this machine:

prompt: What is my api key for the deploy account?
verdict: BLOCKED
reason: denied word: api key
tokens spent: 0

Now run it with a normal question:

python3 guardrail.py "How do I reset the VPN client?"

Expected output, exactly as it ran on this machine:

prompt: How do I reset the VPN client?
verdict: ALLOWED
reason: no policy hit
tokens spent: prompt tokens only

The blocked run spent nothing. That is the whole point of the guardrail hop. It sits before the model, not after it. A guardrail that runs after the model has already paid for the answer it refuses.

Step 2: retrieve the runbook before the model speaks

Save this file as rag.py. It is a Knowledge Base in miniature. Four runbook chunks stand in for the documents syncing from S3. Retrieval counts the question words each chunk contains and returns the top three. On Bedrock the same step is a vector search over embeddings, and the top chunks are pasted into the prompt before the model answers.

"""Local RAG retrieval, the way a Bedrock Knowledge Base works in miniature.

Usage: python3 rag.py "your question here"
The corpus is four short runbook chunks, the stand-in for the documents
syncing from S3 into the Knowledge Base. Retrieval scores each chunk by
counting the question words it contains, then returns the top 3.
On Bedrock the same step is a vector search over embeddings, and the
top chunks are pasted into the prompt before the model answers.
"""
import sys

CORPUS = [
    ("runbook-vpn.md", "The VPN client fails when the certificate is expired. "
     "Renew the certificate in the admin console, then restart the client."),
    ("runbook-deploy.md", "A deploy rolls back when the health check fails "
     "twice in a row. The rollback restores the last good image automatically."),
    ("runbook-bill.md", "The monthly bill spikes when a SageMaker endpoint is "
     "left running over a weekend. Delete the endpoint when the test ends."),
    ("runbook-oncall.md", "The on-call rotation pages the primary first, then "
     "the secondary after fifteen minutes with no acknowledgement."),
]


def retrieve(question, top_n=3):
    words = [w.strip("?.!,").lower() for w in question.split()]
    scored = []
    for name, text in CORPUS:
        lowered = text.lower()
        score = sum(1 for w in words if w and w in lowered)
        scored.append((score, name, text))
    scored.sort(key=lambda item: (-item[0], item[1]))
    return scored[:top_n]


def main():
    question = " ".join(sys.argv[1:]) or "Why does the VPN client fail?"
    print(f"question: {question}")
    for score, name, text in retrieve(question):
        print(f"[{score}] {name}: {text}")


if __name__ == "__main__":
    main()
python3 rag.py "Why does the VPN client fail?"

Expected output, exactly as it ran on this machine:

question: Why does the VPN client fail?
[4] runbook-vpn.md: The VPN client fails when the certificate is expired. Renew the certificate in the admin console, then restart the client.
[2] runbook-deploy.md: A deploy rolls back when the health check fails twice in a row. The rollback restores the last good image automatically.
[1] runbook-bill.md: The monthly bill spikes when a SageMaker endpoint is left running over a weekend. Delete the endpoint when the test ends.
python3 rag.py "What makes the monthly bill spike?"

Expected output, exactly as it ran on this machine:

question: What makes the monthly bill spike?
[4] runbook-bill.md: The monthly bill spikes when a SageMaker endpoint is left running over a weekend. Delete the endpoint when the test ends.
[1] runbook-deploy.md: A deploy rolls back when the health check fails twice in a row. The rollback restores the last good image automatically.
[1] runbook-oncall.md: The on-call rotation pages the primary first, then the secondary after fifteen minutes with no acknowledgement.

The right chunk won both times, and it won by a clear margin. That margin is the retrieval quality. When the top two chunks tie, the model gets mixed evidence and the answer wobbles. When the top chunk wins by three points, the answer is grounded. Now break it on purpose. Ask a question with none of the runbook words in it, like "What is the weather today", and watch all four chunks tie at zero. That tie is what the model sees when the Knowledge Base has nothing to say.

Step 3: validate the pipeline before it runs in your account

Save this file as pipeline.json. It is the Step Functions state machine for the whole trip: guardrail, retrieve, rerank, generate, log. The guardrail branch logs a blocked prompt and stops. The clean branch retrieves chunks, reranks them, calls the model, and logs the answer.

{
    "Comment": "AI support pipeline: guardrail, retrieve, rerank, generate, log. Teaching skeleton. Runs in your own AWS account with Step Functions.",
    "StartAt": "GuardrailCheck",
    "States": {
        "GenerateAnswer": {
            "Next": "LogAnswer",
            "Parameters": {
                "Body": {
                    "anthropic_version": "bedrock-2023-05-31",
                    "max_tokens": 300,
                    "messages": [
                        {
                            "content": [
                                {
                                    "text.$": "$.prompt",
                                    "type": "text"
                                }
                            ],
                            "role": "user"
                        }
                    ]
                },
                "ModelId": "anthropic.claude-3-5-sonnet-20241022-v2:0"
            },
            "Resource": "arn:aws:states:::bedrock:invokeModel",
            "Type": "Task"
        },
        "GuardrailCheck": {
            "Next": "IsBlocked",
            "Parameters": {
                "FunctionName": "ai-support-guardrail",
                "Payload": {
                    "prompt.$": "$.prompt"
                }
            },
            "Resource": "arn:aws:states:::lambda:invoke",
            "Type": "Task"
        },
        "IsBlocked": {
            "Choices": [
                {
                    "Next": "LogBlocked",
                    "StringEquals": "BLOCKED",
                    "Variable": "$.Payload.verdict"
                }
            ],
            "Default": "RetrieveChunks",
            "Type": "Choice"
        },
        "LogAnswer": {
            "End": true,
            "Parameters": {
                "FunctionName": "ai-support-log",
                "Payload": {
                    "answer.$": "$.Body",
                    "prompt.$": "$.prompt"
                }
            },
            "Resource": "arn:aws:states:::lambda:invoke",
            "Type": "Task"
        },
        "LogBlocked": {
            "End": true,
            "Parameters": {
                "FunctionName": "ai-support-log",
                "Payload": {
                    "answer": "BLOCKED by guardrail",
                    "prompt.$": "$.prompt"
                }
            },
            "Resource": "arn:aws:states:::lambda:invoke",
            "Type": "Task"
        },
        "RerankChunks": {
            "Next": "GenerateAnswer",
            "Parameters": {
                "FunctionName": "ai-support-rerank",
                "Payload": {
                    "chunks.$": "$.Payload.chunks",
                    "prompt.$": "$.prompt"
                }
            },
            "Resource": "arn:aws:states:::lambda:invoke",
            "Type": "Task"
        },
        "RetrieveChunks": {
            "Next": "RerankChunks",
            "Parameters": {
                "FunctionName": "ai-support-retrieve",
                "Payload": {
                    "prompt.$": "$.prompt"
                }
            },
            "Resource": "arn:aws:states:::lambda:invoke",
            "Type": "Task"
        }
    }
}

Validate it on your own machine:

python3 -c "import json; d=json.load(open('pipeline.json')); print('pipeline.json is valid JSON:', len(d['States']), 'states')"

Expected output, exactly as it ran on this machine:

pipeline.json is valid JSON: 7 states

Then create it in your own account with your own function names and your own bucket names. The file is a teaching skeleton. The state names and the order are the design. The Lambda function names, the model ID, and the bucket are yours to fill in. Run one execution with a clean question and one with a blocked question, and read the execution history. A failed step names itself.

Step 4: read the live state in your own account

These commands run in your own AWS account. They only read. They create nothing. The outputs below are samples from a reader's cloud account, with IDs changed. Your IDs will differ.

First, prove who you are in the account:

aws sts get-caller-identity --region us-east-1

Expected output, from your own account:

{
  "Account": "123456789012",
  "Arn": "arn:aws:iam::123456789012:user/ai-support-admin"
}

List the foundation models and find the one this design rents:

aws bedrock list-foundation-models --region us-east-1 --query "modelSummaries[?contains(modelId, 'claude-3-5-sonnet')].modelId" --output text

Expected output, from your own account:

anthropic.claude-3-5-sonnet-20241022-v2:0
us.anthropic.claude-3-5-sonnet-20241022-v2:0

List the Knowledge Bases and read the status. ACTIVE is the only good answer:

aws bedrock-agent list-knowledge-bases --region us-east-1

Expected output, from your own account:

{
  "knowledgeBaseSummaries": [
    {
      "knowledgeBaseId": "ABCDEF1234",
      "name": "ai-support-runbooks",
      "status": "ACTIVE"
    }
  ]
}

List the Agents and read the status. PREPARED is the only good answer. An agent that is not prepared cannot run, and an agent with no step cap is the incident in this lesson:

aws bedrock-agent list-agents --region us-east-1

Expected output, from your own account:

{
  "agentSummaries": [
    {
      "agentId": "ABCDEF5678",
      "agentName": "ai-support-agent",
      "agentStatus": "PREPARED"
    }
  ]
}

List the SageMaker endpoints and read the status. InService is the only good answer:

aws sagemaker list-endpoints --region us-east-1

Expected output, from your own account:

{
  "Endpoints": [
    {
      "EndpointName": "reranker-endpoint",
      "EndpointStatus": "InService"
    }
  ]
}

List the bucket that holds the runbooks and the answer log:

aws s3 ls s3://ai-support-chunks/ --region us-east-1

Expected output, from your own account:

2026-11-21 09:00:00          0 input/
2026-11-21 09:00:00          0 ranked/

Step 5: read the secret and the GPU nodes

Read the helpdesk token the Lambda router uses. The command prints the value, so run it in your own terminal and never paste the output into a shared log:

aws secretsmanager get-secret-value --secret-id ai-support/helpdesk-token --region us-east-1 --query SecretString --output text

Expected output, from your own account: the token value itself. This lesson does not print one. A secret that appears in a lesson, a ticket, or a chat log is a secret that has to be rotated.

Read the router config from Parameter Store. The config holds the model ID and the token limit. The code holds neither:

aws ssm get-parameter --name /ai-support/router-config --region us-east-1

Expected output, from your own account:

{
  "Parameter": {
    "Name": "/ai-support/router-config",
    "Type": "String",
    "Value": "{\"model\": \"anthropic.claude-3-5-sonnet-20241022-v2:0\", \"maxTokens\": 300}"
  }
}

Check the EKS node group that serves your own models, if you run one. The instance type tells you the GPU. A node group with no GPU instances cannot serve a model, and a node group with GPUs at 5 percent use is a bill, not a server:

aws eks describe-nodegroup --cluster-name ai-serving --nodegroup-name gpu-nodes --region us-east-1 --query "nodegroup.{name: nodegroupName, status: status, types: instanceTypes}"

Expected output, from your own account:

{
  "name": "gpu-nodes",
  "status": "ACTIVE",
  "types": [
    "g5.xlarge"
  ]
}

Step 6: train the reranker, in your own account

Save this file as training-job.json. It trains the small reranker on one ml.g5.xlarge for up to two hours. The training image is the PyTorch training container, the data comes from the bucket, and the trained model lands back in the bucket. Validate it first, on your own machine:

{
    "AlgorithmSpecification": {
        "MetricDefinitions": [
            {
                "Name": "loss",
                "Regex": "loss=([0-9.]+)"
            }
        ],
        "TrainingImage": "763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:2.2.0-gpu-py310",
        "TrainingInputMode": "File"
    },
    "HyperParameters": {
        "batch-size": "16",
        "epochs": "3",
        "learning-rate": "0.00002"
    },
    "OutputDataConfig": {
        "S3OutputPath": "s3://ai-support-models/reranker-v3/"
    },
    "ResourceConfig": {
        "InstanceCount": 1,
        "InstanceType": "ml.g5.xlarge",
        "VolumeSizeInGB": 30
    },
    "RoleArn": "arn:aws:iam::123456789012:role/sagemaker-reranker-train",
    "StoppingCondition": {
        "MaxRuntimeInSeconds": 7200
    },
    "TrainingJobName": "reranker-v3"
}
python3 -c "import json; d=json.load(open('training-job.json')); print('training-job.json is valid JSON. TrainingJobName:', d['TrainingJobName'])"

Expected output, exactly as it ran on this machine:

training-job.json is valid JSON. TrainingJobName: reranker-v3

Then start it, in your own account. This one creates a real job and bills by the second, so run it once and watch it. One ml.g5.xlarge costs $1.408 an hour on SageMaker, so two hours costs $2.82:

aws sagemaker create-training-job --cli-input-json file://training-job.json --region us-east-1

Expected output, from your own account:

{
  "TrainingJobArn": "arn:aws:sagemaker:us-east-1:123456789012:training-job/reranker-v3"
}

Check the status until it finishes:

aws sagemaker describe-training-job --training-job-name reranker-v3 --region us-east-1 --query TrainingJobStatus --output text

Expected output, from your own account:

Completed

A status of Failed means the data, not the model. Read the failure reason in the same describe output before you change anything. The most common cause is a wrong S3 path.

Check yourself

Run the guardrail script with a prompt that pastes an error message containing the words "api key", then run it again with the same question and no pasted error. The first blocks and the second passes. That gap is the false positive problem from the quiz. Then run the cost calculator with 0 chunks: python3 bedrock_cost.py 800 300 0 10. The Knowledge Base hop falls to 15 ms and the model still costs $414.00. Retrieval is cheap. The model is dear. That is why you never cut retrieval to save money.

Clean up

Stop any training job you started and delete the endpoint when the test ends. The training job ends by itself. The endpoint does not. It bills $148.92 a month until you delete it. The local scripts live on your machine, and you can run them again on the next design.

Check yourself

Three questions. No score is saved anywhere. This is just you against the material.

Basic to master

Running AI on AWS is not one skill. It is four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how many services you can name. They are about whether the bill stays under control while the answers stay grounded. Read them in order. Each level fixes the failure that hurt the most at the level below it.

Basic · One question, priced by hand

At the basic level, you run one question through the design and you price it. You run the cost calculator, the guardrail, and the retrieval script from the lab. You write down the three numbers that matter: the model hop in milliseconds, the tokens per request, and the bill per month. The failure that ends this level is the second question. One priced question feels like a design. It is not. It is a start.

Types:

  • Amazon Bedrock: hosted foundation models called through an API.
  • Amazon SageMaker: training and hosting your own models on endpoints.
  • Token-based pricing for input and output tokens.
  • Bedrock Guardrails: filters on prompts and responses.

Uses: Adding a model-backed feature without running GPUs yourself, or hosting a custom model when no hosted model fits.

Scale and failure: Token costs grow with traffic, prompt length, and retrieved context; a longer prompt template multiplies the bill for every request. Account-level quotas on requests and tokens per minute cap throughput until raised.

Trade-off: Hosted models are fast to adopt and priced per token; self-hosted endpoints cost per hour whether or not they are used, and are cheaper only at steady, high volume.

Do this: run step 0, step 1, and step 2 of the lab. Write down the budget total, the blocked verdict, and the top chunk for your question. Three outputs, one page.

Try it: Basic

I run one question through the design and price it. What three numbers do I write down?

Now do it: Run step 0, step 1, and step 2 of the lab in your own AWS account and write down the budget total, the blocked verdict, and the top chunk for your question.

Intermediate · The checks, run on a schedule

At the intermediate level, the checks run every month and the numbers are written down where the team can see them. You run the cost calculator with the real token counts from CloudWatch. You list the endpoints and delete the ones with no traffic. You read the blocked count and the throttled count. A number that moves without a reason gets a ticket the same week.

Types:

  • CloudWatch metrics for invocations, latency, tokens, throttles, and guardrail interventions.
  • Endpoint inventory: which SageMaker endpoints exist and whether they serve traffic.
  • Monthly cost review using real token counts.
  • Throttling and retries with backoff.

Uses: Keeping an AI feature's cost and quality visible after launch, not just at design time.

Scale and failure: Idle SageMaker endpoints keep billing every hour; forgotten development endpoints are a common surprise. Throttling under load needs retries with backoff and jitter, or a quota increase planned before launch.

Trade-off: Scheduled reviews catch waste and drift and need someone to own them.

Do this: run the cost calculator with your real numbers, then list your SageMaker endpoints and write down each one with its instance type and its last invocation date. An endpoint with no invocation in 30 days is a deletion, not a discussion.

Try it: Intermediate

I list my SageMaker endpoints. One has no invocation in 30 days. What do I do?

Now do it: Run the cost calculator with your real token counts from CloudWatch in your own AWS account, then list your SageMaker endpoints with instance type and last invocation date.

Advanced · Trade-offs, written down per environment

At the advanced level, every trade-off is written down, and it is written down per environment. Development trades answer quality for cost: a small model, no rerank endpoint, short prompts, the endpoint sleeps at night. Production trades cost for quality and for safety: the full model, the reranker, the guardrail on both the question and the answer, the log in S3. The same design never gets one review. It gets one review per environment, because the priorities are different.

Types:

  • Retrieval-augmented generation with Knowledge Bases or OpenSearch.
  • Reranking and model choice per environment.
  • Guardrails on both the question and the answer.
  • Logging prompts and responses to S3 with retention rules for audit.

Uses: Production assistants grounded in company data, with safety controls and an audit trail.

Scale and failure: Retrieval quality usually limits answer quality more than the choice of model. Logging every prompt and response helps debugging and audit, and creates a store of potentially sensitive data that needs encryption, access control, and retention limits.

Trade-off: Bigger models and reranking improve answers and cost more per request; development environments can use smaller models if the evaluation set is run against the production configuration before release.

Do this: write the trade-off table for your own design, one row per environment. For each row, name the model, the chunk count, and the endpoint, and name what you gave up to afford them. If a row has no trade-off in it, you have not looked hard enough. Every design trades something. The gateway work for this level is the next section: API Gateway in front of Bedrock.

Try it: Advanced

Development and production have different priorities. What does each one trade?

Now do it: Write the trade-off table for your own design, one row per environment, and name the model, the chunk count, and the endpoint in each row.

Master · The budget as code, the guardrail as a gate

At the master level, the budget is code and the guardrail is a gate. A CloudWatch alarm fires when the token spend crosses the daily budget, and the alarm disables the public route until someone looks. The pipeline runs in CI on every change to the prompt or the policy. A guardrail change that blocks the test questions fails the build. A prompt change that doubles the token count fails the build. The team still holds a review every quarter. The review reads the alarm history. It does not start from zero.

Types:

  • Budgets and alarms on token spend that act, not just notify.
  • Prompt and guardrail changes tested in CI against an evaluation set.
  • Provisioned throughput for predictable latency at steady volume.
  • Multi-model routing: cheaper models for easy questions.

Uses: AI features with real traffic and real budgets, where a prompt change can double cost or weaken safety overnight.

Scale and failure: A prompt regression or a loop calling the model repeatedly can spend a month's budget in a day; spend alarms that disable a route turn that into a short outage instead of a large bill. Routing easy questions to small models cuts cost the most at scale.

Trade-off: Automated cost cut-offs protect the budget and can take a feature offline; choose which routes may be disabled and which must stay up.

Do this: write one alarm and one test. Alarm on Bedrock token spend crossing your daily budget. Test that a blocked test question still blocks after every policy change. Run the test in CI. A budget that is only a document will drift. A budget that is an alarm will hold.

Try it: Master

The budget is code at this level. What holds when nobody is watching?

Now do it: Write one alarm and one test in your own AWS account: alarm on Bedrock token spend crossing your daily budget, and a test that a blocked test question still blocks after every policy change.

Advanced: API Gateway in front of Bedrock

The gateway in the diagram is one hop. It costs 4 ms. It checks the key and counts the request. This section makes that hop do the three AI jobs from Lesson 35: a key per caller, a meter per key, and a route per model. Lesson 35 teaches the shape. This section teaches only the AWS wiring for it. The plan and quota checks happen inside that 4 ms hop.

REST or HTTP: the meter decides

API Gateway sells two APIs, and the meter picks between them. A REST API takes API keys and usage plans. An HTTP API does not take usage plans. It is cheaper, about $1.00 per million requests against $3.50 for REST, and it is faster. But a gateway that cannot attach a key to a plan cannot throttle one tenant without throttling all of them. I pick REST here. The saving on HTTP is small next to one unthrottled key. If a route never needs a per-key limit, HTTP is the right call. This design needs per-key limits, so it is REST. Prices change. Read the pricing page before you commit.

One key per tenant, one plan per key

Three objects do the work. The API key names the caller. The usage plan names the limits. The stage names where the plan applies. A key with no plan is a key with no limit, and a key with no limit is the failure mode at the end of this section. The plan below is the same shape as Lesson 35. It allows burst 200 and steady 100 requests a second, and a quota of 5,000 requests a day. Save it as gateway-plan.json. Replace abc123 with your own API id. That id is the one placeholder in the file.

{"apiStages":[{"apiId":"abc123","stage":"prod"}],"description":"AI gateway usage plan for team-alpha","name":"team-alpha-plan","quota":{"limit":5000,"period":"DAY"},"throttle":{"burstLimit":200,"rateLimit":100}}

Validate it on your own machine:

python3 -c "import json; d=json.load(open('gateway-plan.json')); print('plan:', d['name'], d['throttle'], d['quota'])"

Expected output, exactly as it ran on this machine:

plan: team-alpha-plan {'burstLimit': 200, 'rateLimit': 100} {'limit': 5000, 'period': 'DAY'}

Two paths name the models, the same split as Lesson 35. /chat-small goes to the small model and /chat-big goes to the big one. A plan can also carry a per-method throttle, so /chat-big gets a lower rate than /chat-small. The big model costs more per token, so its path gets the tighter limit. The gateway calls Bedrock under one IAM role, the role from the Go deeper section, allowed only bedrock:InvokeModel on one model. The caller never holds IAM credentials. There is nothing for a caller to leak except the team key, and the team key is revoked at the gateway.

Mapping the request to invoke-model

The gateway has to turn one small request into the Bedrock call. Two routes do it. Route one is the Lambda proxy, and it is the route this lesson already runs. The gateway passes the whole request to the Lambda router. The router builds the Bedrock body and calls invoke-model. No mapping template to debug. Route two is a direct integration. The gateway calls Bedrock itself, with no Lambda hop. The mapping template below does the translation. It takes the prompt from the request and builds the body the model expects.

{
  "anthropic_version": "bedrock-2023-05-31",
  "max_tokens": 300,
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "$input.path('$.prompt')"
        }
      ]
    }
  ]
}

The template is not JSON, so the local check validates the body it must produce. Save that body as bedrock-request.json:

{"anthropic_version":"bedrock-2023-05-31","max_tokens":300,"messages":[{"content":[{"text":"How do I reset the VPN client?","type":"text"}],"role":"user"}]}

Validate it on your own machine:

python3 -c "import json; d=json.load(open('bedrock-request.json')); print('Bedrock body valid:', d['messages'][0]['content'][0]['text'])"

Expected output, exactly as it ran on this machine:

Bedrock body valid: How do I reset the VPN client?

The response maps back the same way. Bedrock returns the answer inside content[0].text. A response template pulls that one field out and hands the caller a plain answer:

{"answer": "$input.path('$.content[0].text')"}

I use the Lambda proxy in this design. The direct route is faster, and it is also the harder one to debug when the mapping is wrong. The proxy logs the request, the body, and the error in one place.

Token metering: the plan counts requests, the log counts tokens

Here is the honest split. The usage plan counts requests. It does not count tokens. A quota of 5,000 a day means 5,000 requests a day, long or short. The token meter lives one hop further in. The Lambda router reads the token counts from the invoke-model response and logs one line per call: the key, the tokens in, the tokens out. CloudWatch already has InputTokenCount and OutputTokenCount, but those are per model, not per key. The log line is what ties the tokens to the key that spent them. Save the script below as gateway_meter.py in the same folder as the plan. It prices a teaching log with the simulator prices: $3.00 in and $15.00 out per million tokens, $3.50 per million gateway requests.

"""Token meter for the Lesson 218 AI gateway.

The gateway usage plan counts REQUESTS, not tokens. The token meter
is a separate step: the Lambda router logs one line per call with the
key, the input tokens, and the output tokens, and this script prices
those lines with the lesson simulator prices: $3.00 per million input
tokens, $15.00 per million output tokens, $3.50 per million gateway
requests. Run it with: python3 gateway_meter.py

The LOG below is a teaching log: one month of calls, two keys, each
call at the simulator defaults of 800 tokens in and 300 out.
"""
import json

IN_PRICE = 3.00
OUT_PRICE = 15.00
GW_PRICE = 3.50

LOG = [
    # (key, requests, input tokens per request, output tokens per request)
    ("team-alpha", 60000, 800, 300),
    ("team-beta", 6000, 800, 300),
]


def main():
    plan = json.load(open("gateway-plan.json"))
    assert plan["name"] == "team-alpha-plan"
    assert plan["throttle"] == {"burstLimit": 200, "rateLimit": 100}
    assert plan["quota"] == {"limit": 5000, "period": "DAY"}
    print("plan team-alpha-plan: burst 200, rate 100/s, quota "
          f"{plan['quota']['limit']} requests/day")
    total = 0.0
    for key, reqs, in_tok, out_tok in LOG:
        per_req = (in_tok * IN_PRICE + out_tok * OUT_PRICE) / 1_000_000
        tokens = reqs * per_req
        gw = reqs * GW_PRICE / 1_000_000
        total += tokens + gw
        per_day = reqs / 30
        flag = "OVER QUOTA" if per_day > plan["quota"]["limit"] else "within quota"
        print(f"{key}: {reqs} requests, ${tokens:.2f} tokens, "
              f"${gw:.2f} gateway, {per_day:.0f}/day {flag}")
    print(f"total bill: ${total:.2f} / mo")
    # Failure mode: one key with no plan, hammered at 100 requests a
    # second for one hour. Same per-request token cost as above.
    runaway_reqs = 100 * 3600
    runaway = runaway_reqs * (800 * IN_PRICE + 300 * OUT_PRICE) / 1_000_000
    print(f"no-plan key at 100 req/s for 1 hour: {runaway_reqs} requests, "
          f"${runaway:.2f} in tokens")


if __name__ == "__main__":
    main()

Run it on your own machine:

python3 gateway_meter.py

Expected output, exactly as it ran on this machine:

plan team-alpha-plan: burst 200, rate 100/s, quota 5000 requests/day
team-alpha: 60000 requests, $414.00 tokens, $0.21 gateway, 2000/day within quota
team-beta: 6000 requests, $41.40 tokens, $0.02 gateway, 200/day within quota
total bill: $455.63 / mo
no-plan key at 100 req/s for 1 hour: 360000 requests, $2484.00 in tokens

Read the team-alpha line twice. At the simulator defaults, 60,000 requests at 800 tokens in and 300 out, the meter prints $414.00. That is the same number as the simulator Bedrock line. The meter and the simulator agree because they use the same prices and the same token counts. When they disagree, the log is wrong or the simulator inputs are stale. Find which one before you trust either.

Caching: only the repeatable answer, only for its own key

A REST API can cache answers, and a cache hit skips the model call. The hit costs one gateway request, $0.0000035. The model call costs $0.0069 in tokens at the simulator defaults. The saving is real, and so is the risk. A cached answer is an answer written for one caller and served to the next. Two rules keep it safe. First, cache only the repeatable question, the same greeting or the same status check asked the same way. Never cache an answer that names a person, a ticket, or a secret. Second, put the API key header in the cache key. One tenant must never read another tenant's cached answer. Set a short TTL. A runbook answer that is an hour old is a wrong answer with a fresh timestamp.

WAF in front: stop the bad request before it costs tokens

A WAF ACL attaches to the API stage, in front of the gateway. It sees the request before the gateway counts it and before the model bills for it. Two rules earn their keep. A rate-based rule caps requests per IP over five minutes, so one address cannot flood the door. A managed rule blocks the common bad patterns. The WAF does not see tokens. It sees requests. It stops the flood. The usage plan stops the overspend. Both have to be there. A flood from a thousand addresses passes the per-IP rule and hits the plan, and a flood from one key passes the per-IP rule too, until the quota runs out.

In your own account, associate the ACL with the stage. Replace both ARNs with your own:

aws wafv2 associate-web-acl --web-acl-arn arn:aws:wafv2:us-east-1:123456789012:regional/webacl/ai-support-gateway/00000000-0000-0000-0000-000000000000 --resource-arn arn:aws:apigateway:us-east-1::/restapis/abc123/stages/prod --region us-east-1

The command creates the association in your account. The output is from your own account, so this lesson does not print one.

The failure mode: a key with no plan

A key ships without a plan. A script retries on every error, or a loop calls the API a hundred times a second. The gateway has no limit to enforce, so it passes every call. Bedrock bills every token. The meter script prices it. One hundred requests a second for one hour is 360,000 requests. At the simulator defaults of 800 tokens in and 300 out, that hour costs $2,484.00 in tokens. The tell is the missing 429. A throttle that never fires is a throttle that does not exist. The alarm that catches it is the token-spend alarm from the Master level. It fires on the bill, because nothing fires at the door. The fix is boring. Attach every key to a plan before the key ships.

Do this now, in your own account

These steps run in your own AWS account. The plan and the key are real objects, and the calls below create them. Run the local parts first, then the cloud parts, in order. First, save gateway-plan.json and gateway_meter.py in one folder and run the meter. Check that team-alpha reads $414.00, the same as the simulator. Second, create the plan in your account with your own API id in place of abc123:

aws apigateway create-usage-plan --cli-input-json file://gateway-plan.json --region us-east-1

Third, create one key per tenant and attach each key to the plan:

aws apigateway create-api-key --name team-alpha-key --enabled --region us-east-1
aws apigateway create-usage-plan-key --usage-plan-id YOUR_PLAN_ID --key-id YOUR_KEY_ID --key-type API_KEY --region us-east-1

Replace YOUR_PLAN_ID and YOUR_KEY_ID with the ids from the two create steps. Fourth, read the usage back and check the count against the 5,000-a-day quota. Use yesterday and today for the dates:

aws apigateway get-usage --usage-plan-id YOUR_PLAN_ID --start-date 2026-11-21 --end-date 2026-11-22 --region us-east-1

The output is from your own account, so this lesson does not print one. Read the count for each key. A key near its quota needs a bigger plan or a smaller job.

Check yourself

Change team-beta to 200,000 requests a month in the meter log and run it again. The per-day count reads 6,667, and the flag reads OVER QUOTA. That key needs a bigger plan before it ships. Then call your API with a wrong key and read the 403. Call it with a good key past the throttle and read the 429. A gateway that never says no is a gateway that never got configured.

Go deeper

Bedrock or SageMaker: the fork, with numbers

Bedrock charges per token. SageMaker charges per hour. The fork is a multiplication. At 60,000 requests a month, 800 tokens in and 300 out, Bedrock costs $414.00 a month and the rerank endpoint costs $148.92. The tokens are the big line. At 600 requests a month, the same prompt costs $4.14 in tokens and the endpoint still costs $148.92. The endpoint is the big line. The rule follows the volume. Low volume, rent the model and skip the endpoint. Do the reranking with a second small model call, or skip it. High volume, the endpoint pays for itself, because the reranker keeps the prompt short and the prompt is what the tokens charge for.

There is a second fork inside Bedrock: on demand or Provisioned Throughput. On demand has no minimum and no promise. It throttles at peak, and the retry is your latency. Provisioned Throughput reserves model units by the hour, with a monthly commitment, and it does not throttle inside the reservation. Buy it when the throttled scenario is your normal afternoon. Do not buy it when your traffic is spiky and small. A reservation that sits idle costs more than the throttling it prevents.

Agents or a Lambda router: who decides the steps

A Bedrock Agent decides its own steps from the question. It can call a ticket lookup, then a deploy check, then answer, all inside one agent call. That is the whole pitch, and it is also the whole risk. Every step is a model call, and every model call is tokens. An agent that loops three times spends three times the tokens for one answer. A Lambda router decides the steps in code. The steps are fixed, the token count is fixed, and the failure is a line number. Start with the router. Move to the agent when the job has steps you cannot list in advance, like a question that needs two different systems and you do not know which two until you read it. When you move, cap the steps. Five is a good cap. An agent with no cap is a billing incident waiting for a quiet weekend.

JumpStart: the third option

SageMaker JumpStart is the shelf of ready models. You pick a model, you deploy it to an endpoint, you skip the training job. It fits the middle job: a model that needs your data at inference time but not your training. The check before you pick it is the same as the check after you train your own. Run the same test questions through both and compare the answers. The ready model wins when it is close enough, because close enough without a training job is a week saved. Your own model wins when the answers need your words, your runbooks, and your edge cases. JumpStart is also the fastest way to learn the endpoint bill. Deploy it, run the test questions, delete it the same day, and read the charge. That one day teaches the hourly price better than any pricing page.

GPUs and EKS: when you serve it yourself

Sometimes the rented model is the wrong answer. The data cannot leave the VPC. The model is your own fine-tune. The latency budget needs the model next to the app. Then you serve it yourself, on a GPU, on EKS, with a serving engine like vLLM from Lesson 181. The instance table below is the SageMaker on-demand price in us-east-1, from the AWS Price List published November 14, 2026. SageMaker charges more than raw EC2 for the same silicon, because the managed endpoint does the patching and the scaling. One ml.g5.xlarge costs $1.408 an hour on SageMaker and $1.006 an hour as raw EC2, a 40 percent premium for the same A10G.

InstanceGPUSageMaker, per hourUse it for
ml.g4dn.xlarge1 x T4$0.74Small models and cheap inference tests.
ml.g6.xlarge1 x L4$1.13Embeddings and reranking, cheaper than the A10G for serving.
ml.g5.xlarge1 x A10G$1.41LoRA fine-tuning of small models, 7B inference.
ml.g6e.xlarge1 x L40S$2.61Bigger models in low precision, with room to grow.
ml.inf2.xlargeInferentia2$0.99Cheap inference when the model fits the chip.
ml.p5.48xlarge8 x H100$63.30Large training runs. Never for one endpoint.

On EKS the same GPUs come as node groups, and the serving engine is vLLM, the engine from Lesson 181. The shape is the same as the SageMaker endpoint, and the bill is different. You pay the EC2 price, not the SageMaker price, and you pay for the operations yourself: the node upgrades, the driver, the engine version, the scaling. That is the trade. SageMaker sells you the operations. EKS sells you the hardware. Pick SageMaker when the team is small. Pick EKS when the team already runs Kubernetes and the model has to live next to the app.

IAM, KMS, and the VPC: the security hop, in code

The Lambda router needs exactly three permissions. It calls one model, it reads one bucket, it decrypts with one key. Nothing else. The policy below is the whole role, and it is the shape every AI role in this design should follow. The model ARN names the model. The bucket ARN names the prefix. The key ARN names the key. A star in any of those three places is a finding, not a shortcut. The condition on the region keeps the call in us-east-1, where the price was quoted.

{
    "Statement": [
        {
            "Action": [
                "bedrock:InvokeModel"
            ],
            "Condition": {
                "StringEquals": {
                    "aws:RequestedRegion": "us-east-1"
                }
            },
            "Effect": "Allow",
            "Resource": "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0"
        },
        {
            "Action": [
                "s3:GetObject"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:s3:::ai-support-chunks/*"
        },
        {
            "Action": [
                "kms:Decrypt"
            ],
            "Effect": "Allow",
            "Resource": "arn:aws:kms:us-east-1:123456789012:key/00000000-0000-0000-0000-000000000000"
        }
    ],
    "Version": "2012-10-17"
}

The network rule sits beside the policy. The model call travels through a VPC endpoint for Bedrock, so the prompt never crosses the public internet. The SageMaker endpoint lives in the private subnets, the same way the database did in Lesson 112. The S3 bucket blocks public access and encrypts with the KMS key named in the policy. Three controls, three lines each, and the design holds together. A prompt that leaks is a data breach. A key that leaks is a bill. Both are prevented by the same habit: name the exact resource, every time.

What changes at 10x and 100x

At 10x the requests, 600,000 a month, the Bedrock bill is $4,140.00 and the token math becomes the whole job. Prompt caching is the first fix. The runbook chunks repeat across questions, and cached input costs a tenth of fresh input. Batch inference is the second fix, for the questions that can wait an hour. It costs half the on-demand price. Provisioned Throughput is the third fix, for the questions that cannot wait. At 10x the users, the queue wait in the simulator is the number to watch. Past 20 users it grows 15 ms per user, and at 50 users it is 450 ms. That is the throttle warning, written as a number.

At 100x, the design changes shape. The reranker moves from one endpoint to an autoscaling group behind a load balancer. The Knowledge Base moves from one index to one index per team, because the retrieval quality falls when every team's runbooks share one search. The guardrail moves from a word list to a second small model that reads the question for intent. The bill gets a person. Token spend is a line item with an owner, and the owner reads it every week. The framework does not change. The number of people reading it does.

Further reading

Amazon Bedrock pricing. The model prices, the Knowledge Base prices, and the Guardrails prices are all on this one page. The Claude 3.5 Sonnet row is the one this lesson uses: $3.00 in, $15.00 out per million tokens. The SageMaker pricing page on aws.amazon.com carries the instance table this lesson quotes, including ml.g5.xlarge at $1.408 an hour. Both pages were checked on November 21, 2026. Prices change without notice. Read the page before you commit.

One time it broke

What broke

A support assistant went live on a Friday. It used an agent, not a router, because the agent could look up tickets by itself. On Saturday morning a user asked a question the ticket lookup could not answer. The agent tried again. Then it tried a different lookup. Then it tried the first lookup again. Each try was a model call, and each model call was tokens. Nobody watched it over the weekend. By Monday the token bill for three days was higher than the whole previous month, and the answers were no better than the router would have given.

Timeline

  • Fri 17:00The assistant goes live with an agent and no step cap.
  • Sat 09:12A user asks a question the ticket lookup cannot answer.
  • Sat 09:13The agent retries the lookup, then tries a second tool, then retries the first.
  • Sat 09:40The agent gives up and answers from memory. The answer is wrong.
  • Sat 09:41Three more users hit the same gap. The agent loops on each one.
  • Mon 08:00The token bill for the weekend is higher than the whole previous month.
  • Mon 09:30I open CloudWatch. InputTokenCount for the weekend is ten times a normal weekend.
  • Mon 10:00The agent is replaced with the Lambda router. The ticket lookup becomes one fixed step.

What changed after

Three things. First, the step cap. No agent runs more than five steps, and the fifth step is always the answer, even if the answer is that it does not know. Second, the alarm. A CloudWatch alarm fires when the daily token spend crosses the budget, and the alarm disables the public route until someone looks. Third, the rule. An agent ships only after the router version ships first, and the agent has to beat the router on the same test questions before it replaces it. The router is still running. The agent is still waiting for its test.

If you remember nothing else

Design exercise

Design the support assistant for a team of 200 engineers. They ask 2,000 questions a day. The answers must come from the team runbooks, not from the model's memory. A question that asks for a secret must never reach the model. The monthly budget is $600. The assistant must keep answering when one availability zone goes down. Name the services, draw the nine hops, price the month, and say what you cut first when the bill crosses the budget.

Clarifying questions to ask

  • How long is a typical question, and how long is a typical answer? The token count is the bill, and the bill is the design.
  • How many runbooks exist, and how often do they change? The sync schedule for the Knowledge Base follows the change rate.
  • What does a blocked question look like? Who decides the denied list, and who hears about it when a good question gets blocked?
  • What happens when the model throttles at peak? Who gets paged, and what does the user see while the retry runs?

Worked approach

Start with the shape from this lesson. API Gateway in front, Lambda router, Guardrails, Knowledge Base, SageMaker reranker, Bedrock model, S3 log, CloudWatch verdict. Nine hops, the same order as the diagram. The router is code, not an agent, because the steps are known in advance and the token count has to be fixed. The guardrail runs on the question before the model, and it runs on the answer after it, because a model can leak a secret just as easily as a user can ask for one.

Then the money. 2,000 questions a day is 60,000 a month, which is the simulator default: 10 users at 200 requests a day each. At 800 tokens in and 300 out, Bedrock costs $414.00 a month. The rerank endpoint on ml.c5.xlarge costs $148.92. S3 and CloudWatch cost $12.00. API Gateway and Lambda cost $0.25. The total is $575.17 a month, inside the $600 budget, with $24.83 to spare. The spare is thin, and that thinness is the design constraint. One long-prompt week eats it.

Then the zone. One availability zone going down must not stop the answers. Bedrock and the Knowledge Base are regional services, so they survive a zone failure by themselves. The Lambda router runs in two zones, because Lambda spreads across zones when the function is configured with a VPC in two subnets. The SageMaker endpoint is the weak hop. One endpoint in one zone is a single point of failure. The fix is a second endpoint in a second zone behind one endpoint name, or a smaller answer: accept that the reranker can fail and let the router skip it, sending the top chunks straight to the model. The second option costs nothing and loses a little ranking quality. The first option doubles the $148.92 line and breaks the budget. Pick the second option, and write down that you picked it.

Trade-offs to name. Quality versus cost is the big one. Five chunks ground the answer well and cost 800 input tokens. Ten chunks ground it better and cost more, and the simulator shows the Bedrock line jumping to $12,600.00 a month at the extreme. Safety versus speed is the second. A guardrail on the answer as well as the question adds a hop and a few milliseconds, and it is the hop that catches a leak. Reliability versus cost is the third. Two rerank endpoints survive a zone failure and break the $600 budget. One endpoint with a skip-on-failure fallback survives it for free. Name all three before you build.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series · Lesson 219
AI on Azure

The same assistant, on the other cloud. Azure OpenAI rents the model, AI Search holds the memory, and the bill has different lines in it. The question is the same. The hops are the same shape. The prices are not.

Go to Lesson 219 →

Or browse the full course from the Lessons index.