Lesson 112 taught me GCP. Lesson 218 and Lesson 219 taught me AI on AWS and AI on Azure. This lesson is the same idea on Google's cloud: one question, answered by a model, with a whole pipeline around it. The model is the famous part. It is also the smallest part of the system. Around it sits a function, a secret, a queue, a workflow, a bucket, and a dashboard. I have seen teams buy the model and forget the pipeline. The model answers. The pipeline decides whether the answer is true, whether it arrives on time, and whether the bill makes sense. This lesson walks one question through that pipeline, hop by hop, on GCP.
Difficulty: Advanced · Know this first: Lesson 218, Lesson 219, Lesson 16, Lesson 87
A support team asked me for one thing. A box where a user types a question and gets an answer from the docs. Not from the model's memory. From the docs. The refund window, the shipping rules, the warranty terms. The answers had to quote the doc they came from. If the answer was not in the docs, the box had to say so.
We built it on GCP. A user asks a question. A function takes it. The question waits on a topic while a workflow reads the docs out of a bucket and calls the model on a Vertex AI endpoint. The answer comes back with the token counts, and the numbers go to a dashboard. Nine hops. One question. One grounded answer.
Then the team asked the second question, the one that decides every AI system I have built since. What happens when the secret is wrong, the endpoint is cold, or the quota runs out at noon? This lesson is the map for that too. The happy path is nine hops. The failure paths are the same nine hops, with one of them saying no.
By the end of this lesson you will have run a local stand-in for a Vertex AI endpoint, built the exact request body a Cloud Function sends to it, written the workflow that orders the steps, and worked the latency, cost, and quota numbers with a script you run on your own machine. Then you will have the commands to build the real thing in your own project: the endpoint, the secret, the topic, the TPU training job, and the GKE serving deployment. Two more doors close the lesson: a gateway in front of the endpoint, and Cloud Run serving the model the team runs itself. The cloud steps need your own account, and the lesson says so plainly where they start. Nothing before that step does.
The two panels share the same bucket and the same project, and nothing else. The left panel trains a model. The right panel answers a question with one. Training is a batch job that runs for hours. Serving is a request that must answer in under a second. Teams mix them up when they put training on the serving path, or when they serve from a training job that was never built to answer. Read the two panels left to right, then bottom to top: the bottom line is the one that matters.
The two panels share the bucket and nothing else. Training writes the model artifact. Serving reads it through the endpoint. When a team serves straight from a training job, the first user waits for a training run to finish. When a team trains on the serving endpoint, the users wait for the training. Keep the two panels apart and both of them are simple.
The budget bar above shows three fixed stories. This simulator lets you write your own. It answers one question: how long does one grounded answer take, and what does it cost. You set how many tokens go in, how many come out, how many requests arrive at once, how many docs ground the answer, and whether grounding is on at all. Move a slider and watch the endpoint time, the storage time, the queue wait, the total, and the verdict change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: the prompt is the latency, the queue is the surprise, and grounding is a choice with a price in both directions.
How many tokens go into the model: the docs plus the question. Eight hundred is two short docs and one question. Four thousand is a long prompt with half the manual pasted in. More tokens in means a longer answer time, and a bigger bill.
How many tokens the model may write back. Two hundred is two sentences. One thousand is a full page. Output tokens cost more than input tokens, and each one takes longer to write than one to read.
How many questions arrive at the same time. Eight is a quiet minute. Sixty-four is a launch. The endpoint handles 32 at once without waiting. Past that, each extra request waits 12 ms in the queue.
How many docs the workflow reads from the bucket before it calls the model. Each doc adds 12 ms of storage time, on top of 14 ms fixed. Zero docs means the model answers with nothing to stand on, even when grounding is on.
Checked means the workflow reads the docs from the bucket and puts them in the request. Unchecked means the storage hop is skipped and the model answers from memory. That is 38 ms faster at the defaults, and it is how invented facts get into answers.
Try this: (1) Leave the defaults: 800 tokens in, 200 out, 8 at once, 2 docs, grounding on. The total is 507 ms and 1,000 answers cost $2.00. That is the normal scenario, rebuilt from the knobs. (2) Push the input to 4,000 tokens and the output to 1,000. The endpoint jumps to 1,700 ms, the total to 1,787 ms, and 1,000 answers cost $10.00. Same pipeline, same model. Only the prompt changed. (3) Set the prompt back and push concurrent requests to 64. The queue adds 384 ms, the endpoint takes 804 ms, and the total is 891 ms. The model did not get slower. The line got longer. (4) Set everything back and uncheck grounding. The storage hop drops to zero and the total falls to 469 ms. Thirty-eight milliseconds faster, and the answer is ungrounded. That is the cheapest bad trade in this lesson.
Formulas, in plain terms. Storage ms = 14 plus 12 for each context doc, when grounding is on, else zero. Endpoint ms = 100 plus 0.20 for each input token plus 0.80 for each output token, plus 12 ms of queueing for every concurrent request past 32. Total ms = 49 plus storage plus endpoint, where 49 is the fixed pipeline: client 1, function 14, secret 8, topic 6, workflow 11, monitoring 5, answer 4. Cost per answer = input tokens times $1.25 per million plus output tokens times $5.00 per million. Those are example prices from this lesson, not a quote. Moving a control updates the storage time on hop 6, the endpoint time on hop 7, the budget bar, and the step panel above.
Do this now, in your own terminal. You will run a local stand-in for a Vertex AI endpoint, build the exact request body a Cloud Function sends to it, write the workflow that orders the steps, and work the latency, cost, and quota numbers with a script. The cloud steps need your own project, and they say so plainly where they start. Nothing before that step does.
python3 --version
python3 -c "import yaml; print('yaml ok')"
Versions that ran for this lesson: Python 3.12.3, with PyYAML installed. Any Python 3.10 or newer works. The yaml check should print yaml ok. If it does not, install PyYAML for your Python. The cloud steps need the Google Cloud CLI, gcloud. Nothing before the cloud steps does, and every check below runs with no cloud account and no credentials.
Save this as vertex_mock.py. It answers the same generateContent shape the real endpoint returns: candidates, a text part, and usageMetadata with the token counts. The answer text is canned. The shape is the point, not the model.
import http.server, json
ANSWER = (
"The refund window is 30 days from delivery. "
"See the returns doc, section 2."
)
BODY = {
"candidates": [
{
"content": {
"parts": [{"text": ANSWER}],
"role": "model",
},
"finishReason": "STOP",
}
],
"usageMetadata": {
"promptTokenCount": 812,
"candidatesTokenCount": 96,
"totalTokenCount": 908,
},
}
class Handler(http.server.BaseHTTPRequestHandler):
def do_POST(self):
length = int(self.headers.get("Content-Length", 0))
raw = self.rfile.read(length) if length else b"{}"
try:
req = json.loads(raw)
parts = req["contents"][0]["parts"]
text = parts[0]["text"]
except (KeyError, IndexError, TypeError, ValueError):
self.send_response(400)
self.send_header("Content-Type", "application/json")
self.end_headers()
self.wfile.write(json.dumps({"error": "bad request body"}).encode())
return
if "refund" not in text.lower():
self.send_response(400)
self.send_header("Content-Type", "application/json")
self.end_headers()
self.wfile.write(
json.dumps({"error": "this mock only answers refund questions"}).encode()
)
return
payload = json.dumps(BODY).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(payload)))
self.end_headers()
self.wfile.write(payload)
def log_message(self, *args):
pass
if __name__ == "__main__":
srv = http.server.HTTPServer(("127.0.0.1", 8080), Handler)
print("vertex mock listening on http://127.0.0.1:8080")
srv.serve_forever()
Save the request body as generate-request.json. This is the exact body the function builds in step 3. Docs first, question last.
{
"contents": [
{
"role": "user",
"parts": [
{
"text": "Use only the docs below to answer.\n\nDOC: The refund window is 30 days from delivery.\n\nDOC: Refunds go back to the original payment method.\n\nQUESTION: What is the refund window?"
}
]
}
],
"generationConfig": {
"maxOutputTokens": 200,
"temperature": 0.2
}
}
Start the mock in one shell with python3 vertex_mock.py, then call it from another shell. Expected output, exactly as it ran in this build:
curl -s -X POST http://127.0.0.1:8080/v1/projects/demo/locations/us-central1/publishers/google/models/gemini-2.5-flash:generateContent -H "Content-Type: application/json" -d @generate-request.json
{"candidates": [{"content": {"parts": [{"text": "The refund window is 30 days from delivery. See the returns doc, section 2."}], "role": "model"}, "finishReason": "STOP"}], "usageMetadata": {"promptTokenCount": 812, "candidatesTokenCount": 96, "totalTokenCount": 908}}
Read the three token counts. The prompt cost 812 tokens, the answer cost 96, and the total is 908. That is the bill for one answer, in tokens. Now ask it something it does not know, and read the failure:
curl -s -X POST http://127.0.0.1:8080/v1/projects/demo/locations/us-central1/publishers/google/models/gemini-2.5-flash:generateContent -H "Content-Type: application/json" -d '{"contents":[{"role":"user","parts":[{"text":"hello"}]}]}'
{"error": "this mock only answers refund questions"}
That 400 is the mock saying the question has no grounding. The real endpoint would answer anyway, from memory, and that is the difference. A mock can refuse. A model will not. The grounding has to come from your pipeline, not from the model. Stop the server with Ctrl-C when you are done.
Save this as main.py. It is the Cloud Functions (2nd gen) front door at hop 2, cut down to the part you can run here. The parts that touch Google Cloud are marked in the code. The part you run is build_generate_body: it takes a prompt and a list of docs and returns the body. Run it with python3 main.py.
import json
MODEL = "gemini-2.5-flash"
LOCATION = "us-central1"
def build_generate_body(prompt, docs):
"""Build the generateContent body: docs first, question last.
The model only sees what is in this body. The docs are the grounding.
Leave them out and the model answers from memory, which is the
ungrounded failure in this lesson.
"""
context = "\n\n".join("DOC: " + d for d in docs)
text = "Use only the docs below to answer.\n\n" + context + "\n\nQUESTION: " + prompt
return {
"contents": [
{"role": "user", "parts": [{"text": text}]}
],
"generationConfig": {"maxOutputTokens": 200, "temperature": 0.2},
}
def endpoint_url(project, model=MODEL, location=LOCATION):
return (
"https://%s-aiplatform.googleapis.com/v1/projects/%s/locations/%s"
"/publishers/google/models/%s:generateContent"
% (location, project, location, model)
)
if __name__ == "__main__":
docs = [
"The refund window is 30 days from delivery.",
"Refunds go back to the original payment method.",
]
body = build_generate_body("What is the refund window?", docs)
print(json.dumps(body, indent=2))
print(endpoint_url("demo-project"))
Expected output, exactly as it ran in this build:
{
"contents": [
{
"role": "user",
"parts": [
{
"text": "Use only the docs below to answer.\n\nDOC: The refund window is 30 days from delivery.\n\nDOC: Refunds go back to the original payment method.\n\nQUESTION: What is the refund window?"
}
]
}
],
"generationConfig": {
"maxOutputTokens": 200,
"temperature": 0.2
}
}
https://us-central1-aiplatform.googleapis.com/v1/projects/demo-project/locations/us-central1/publishers/google/models/gemini-2.5-flash:generateContent
Compare the printed body with generate-request.json from step 2. They are the same shape, and the text is the same text. That is the check: the function builds exactly what the endpoint expects. Now change one thing and run it again. Pass an empty docs list and watch the body lose its grounding. The question stays. The docs go. That is the ungrounded request, and it is one line of code away from the grounded one.
Save this as answer-workflow.yaml. It runs the four steps in order: read the secret, read the docs, call the model, write the answer. One step feeds the next. If any step fails, the run stops there and the failure is the step that failed.
main:
params: [args]
steps:
- getSecret:
call: googleapis.secretmanager.v1.projects.secrets.versions.access
args:
name: ${"projects/" + args.project + "/secrets/model-api-key/versions/latest"}
result: secret
- readDocs:
call: googleapis.storage.v1.objects.get
args:
bucket: ${args.bucket}
object: ${args.docObject}
result: docs
- callModel:
call: http.post
args:
url: ${"https://" + args.location + "-aiplatform.googleapis.com/v1/projects/" + args.project + "/locations/" + args.location + "/publishers/google/models/gemini-2.5-flash:generateContent"}
auth:
type: OAuth2
headers:
Content-Type: application/json
body:
contents:
- role: user
parts:
- text: '${"Use only the docs below to answer. DOC: " + docs + " QUESTION: " + args.question}'
generationConfig:
maxOutputTokens: 200
temperature: 0.2
result: answer
- writeAnswer:
call: googleapis.storage.v1.objects.insert
args:
bucket: ${args.bucket}
name: ${"answers/" + args.questionId + ".json"}
body: ${answer.body}
result: stored
- done:
return: ${stored}
Prove it parses, with no cloud account:
python3 -c "import yaml; yaml.safe_load(open('answer-workflow.yaml')); print('workflow YAML parses')"
Expected output, exactly as it ran in this build:
workflow YAML parses
One honest note. My first draft of this file did not parse. The text line held an unquoted ${...} expression with DOC: inside it, and YAML read the colon as a new mapping. The fix was single quotes around the whole expression. A workflow file that does not parse is a pipeline that never runs, and the parse check is the cheapest test in this lesson.
Save this as gcp_ai_math.py and run it with python3 gcp_ai_math.py. It computes the latency, the cost, and the quota numbers from this lesson. No cloud account, no network. Plain arithmetic.
"""gcp_ai_math.py — the three numbers that decide an AI-on-GCP answer.
1. Latency: endpoint ms = 100 + 0.20 x input tokens + 0.80 x output tokens,
plus 12 ms of queueing for every concurrent request past 32.
2. Cost: example prices in this lesson, not a quote: $1.25 per million
input tokens, $5.00 per million output tokens.
3. Quota: the project allows 60 requests a minute to the endpoint.
Past that, the endpoint answers 429.
Run: python3 gcp_ai_math.py
No cloud account, no network. Plain arithmetic from the lesson's numbers.
"""
INPUT_PRICE = 1.25 # dollars per million input tokens
OUTPUT_PRICE = 5.00 # dollars per million output tokens
QUOTA_RPM = 60 # requests per minute allowed to the endpoint
def endpoint_ms(in_tokens, out_tokens, concurrent):
queue = 12 * max(0, concurrent - 32)
return 100 + 0.20 * in_tokens + 0.80 * out_tokens + queue
def answer_cost(in_tokens, out_tokens):
return in_tokens / 1e6 * INPUT_PRICE + out_tokens / 1e6 * OUTPUT_PRICE
def total_ms(in_tokens, out_tokens, concurrent, docs, grounded):
storage = (14 + 12 * docs) if grounded else 0
return 49 + storage + endpoint_ms(in_tokens, out_tokens, concurrent)
print("one answer, defaults (800 in, 200 out, 8 concurrent, 2 docs, grounded):")
print(" endpoint %.0f ms, storage %d ms, total %.0f ms"
% (endpoint_ms(800, 200, 8), 14 + 12 * 2, total_ms(800, 200, 8, 2, True)))
print(" cost per answer $%.3f, per 1,000 answers $%.2f"
% (answer_cost(800, 200), answer_cost(800, 200) * 1000))
print()
print("a long prompt (4,000 in, 1,000 out, 8 concurrent, 2 docs, grounded):")
print(" endpoint %.0f ms, total %.0f ms"
% (endpoint_ms(4000, 1000, 8), total_ms(4000, 1000, 8, 2, True)))
print(" cost per answer $%.3f, per 1,000 answers $%.2f"
% (answer_cost(4000, 1000), answer_cost(4000, 1000) * 1000))
print()
print("a busy minute (800 in, 200 out, 64 concurrent, 2 docs, grounded):")
print(" queue adds %d ms, endpoint %.0f ms, total %.0f ms"
% (12 * (64 - 32), endpoint_ms(800, 200, 64), total_ms(800, 200, 64, 2, True)))
print()
print("grounding off (800 in, 200 out, 8 concurrent, grounded=False):")
print(" storage 0 ms, total %.0f ms — 38 ms faster, and the answer is ungrounded"
% total_ms(800, 200, 8, 2, False))
print()
print("quota check, %d requests a minute allowed:" % QUOTA_RPM)
print(" at 2 requests a second you send 120 a minute, so 60 answers come back")
print(" and 60 come back 429. The fix is a quota raise, not more replicas.")
Expected output, exactly as it ran in this build:
one answer, defaults (800 in, 200 out, 8 concurrent, 2 docs, grounded): endpoint 420 ms, storage 38 ms, total 507 ms cost per answer $0.002, per 1,000 answers $2.00 a long prompt (4,000 in, 1,000 out, 8 concurrent, 2 docs, grounded): endpoint 1700 ms, total 1787 ms cost per answer $0.010, per 1,000 answers $10.00 a busy minute (800 in, 200 out, 64 concurrent, 2 docs, grounded): queue adds 384 ms, endpoint 804 ms, total 891 ms grounding off (800 in, 200 out, 8 concurrent, grounded=False): storage 0 ms, total 469 ms — 38 ms faster, and the answer is ungrounded quota check, 60 requests a minute allowed: at 2 requests a second you send 120 a minute, so 60 answers come back and 60 come back 429. The fix is a quota raise, not more replicas.
Read the last block twice. At 2 requests a second, half the answers fail, and the endpoint is not slow. It is full by policy. That is the quota check doing its job, and it is cheaper than a page at noon.
Save this as vllm-deployment.yaml. It is the GKE serving deployment from Go deeper: two replicas of vLLM on a GPU node pool, behind a ClusterIP Service with no public address. Prove it parses:
python3 -c "import yaml; print(len(list(yaml.safe_load_all(open('vllm-deployment.yaml')))), 'documents in vllm YAML')"
Expected output, exactly as it ran in this build: 2 documents in vllm YAML. One Deployment, one Service. The Service is ClusterIP, so the only callers are inside the cluster. The Vertex endpoint and the function are the public face. This one is not.
Save the alert as monitoring-alert.json. It pages when the endpoint p95 latency stays above 2,000 ms for five minutes. Prove it is valid JSON:
python3 -m json.tool monitoring-alert.json > /dev/null && echo ALERT-JSON-OK
Expected output, exactly as it ran in this build: ALERT-JSON-OK. The threshold is 2,000 ms because the simulator in this lesson says a cold endpoint is 2,400 ms. The alert fires before the cold becomes the normal.
This step needs your own cloud account, and I cannot run it here. There is no cloud account in this build. Everything below is for your account, and the outputs shown are the shape of what your own account returns, not output from this machine. Do not paste them into a report as if they ran here.
Replace PROJECT_ID with your own project in every command. First the pieces the pipeline needs:
gcloud config set project PROJECT_ID gcloud storage buckets create gs://PROJECT_ID-docs --location=us-central1 gcloud secrets create model-api-key --data-file=key.txt gcloud pubsub topics create answer-jobs gcloud functions deploy answer-intake --gen2 --runtime=python312 --region=us-central1 --source=. --entry-point=answer_intake --trigger-http gcloud workflows deploy answer-flow --source=answer-workflow.yaml --location=us-central1
Then the endpoint. Create it, deploy the model from Model Garden onto it, and keep one replica warm so the first user never pays the cold start:
gcloud ai endpoints create --display-name=answer-endpoint --region=us-central1 gcloud ai endpoints deploy-model ENDPOINT_ID --model=MODEL_ID --display-name=gemini-2-5-flash --machine-type=n1-standard-4 --min-replica-count=1 --max-replica-count=4 --region=us-central1
Then the training shape and the GKE shape. List the accelerator types first, because the names change and the list in your project is the truth:
gcloud compute tpus accelerator-types list --zone=us-central1-a gcloud compute tpus tpu-vm create train-tpu --zone=us-central1-a --accelerator-type=v5litepod-8 --version=tpu-ubuntu2204-base gcloud compute instances create gpu-train --zone=us-central1-a --machine-type=n1-standard-4 --accelerator=type=nvidia-tesla-t4,count=1 --maintenance-policy=TERMINATE --restart-on-failure gcloud container clusters create ai-serving --zone=us-central1-a --num-nodes=2 --machine-type=n1-standard-4 kubectl apply -f vllm-deployment.yaml
Prove the endpoint answers, from your account:
gcloud ai endpoints predict ENDPOINT_ID --region=us-central1 --json-request=generate-request.json
Expected output, the shape of it: a predictions list with one entry, the answer text inside it, and the token counts beside it. The same shape the mock returned in step 2, from the real model this time. When you are done, tear it down: delete the endpoint, the TPU, the GPU VM, and the cluster. An endpoint left running bills by the hour, and a TPU left running bills by the hour too.
Change one thing and run it again. Set maxOutputTokens to 50 in main.py and run python3 main.py again. The body still builds. Then run python3 gcp_ai_math.py and read the cost line for a long prompt at 50 output tokens instead of 1,000. The answer gets shorter, the cost falls, and the endpoint gets faster. The config change is one number. The difference is a page of output the user never asked for.
Three questions. No score is saved anywhere — this is just you against the material.
AI on GCP is not one thing. It is four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how many models you own. They are about how much of the pipeline around the model is deliberate: the secret, the queue, the grounding, and the numbers. Read them in order. Each level fixes the failure that hurt the most at the level below it.
At the basic level, one Cloud Function calls the model and returns the answer. The key sits in an environment variable. There is no queue, no workflow, and no grounding. The function pastes the question into the request and sends it. It works. The user gets an answer. That is the whole system, and it is how every AI service starts.
Do this: run the mock lab in this lesson, then call the mock with the docs removed from the request body. Read the answer. It is confident, it is fast, and it is made up. That is the failure the next level fixes, and you have just seen it with your own eyes.
Uses: A demo for one stakeholder. A first version built to answer one question: does the model answer at all. I use this level to learn, never to serve real users.
Failure modes: The key leaks, because it lives in an environment variable somebody pastes into a chat. The answer invents facts, because nothing grounds it. A slow model call holds the user's connection open, because there is no queue between the user and the model.
Try it: Basic
At the basic level, where does the key live and what does the model read before it answers?
Now do it: Run the mock lab in this lesson, then call the mock with the docs removed from the request body and read the answer. It is confident, fast, and made up.
At the intermediate level, the pipeline is the system and the model is one hop in it. The key lives in Secret Manager and the function's service account has one role on that one secret. The question goes on a Pub/Sub topic and the function answers at once. Workflows runs the steps in order: secret, docs, model, write. The docs go into the request before the model sees the question. The numbers go to Cloud Monitoring. This is the level this lesson teaches in full.
Do this now: run the lab in this lesson, then set grounding off in the simulator and watch the total fall by 38 ms. Then read the try-this text again. Thirty-eight milliseconds faster, and the answer is ungrounded. You just priced the trade, and the price is the lesson.
Uses: A small team serving a real support box. An internal tool where a wrong answer is an annoyance, not a loss. Any system where the callers are people you can talk to when something breaks.
Failure modes: The secret rotates and the function keeps reading a copy from memory, so the new key never reaches the running instances. The quota runs out at noon, because nobody asked what the default was. The endpoint goes cold overnight and the first user of the morning waits two and a half seconds.
Try it: Intermediate
I set grounding off in the simulator and the total falls by 38 ms. Is that a good trade?
Now do it: Run the lab in this lesson, then set grounding off in the simulator and watch the total fall by 38 ms. Then read the try-this text again and decide if the trade is worth it.
At the advanced level, the team owns both doors. The Vertex endpoint serves the managed model, and a vLLM deployment on GKE serves the model the team runs itself. The endpoint also gets a front door. API Gateway stands in front of it with one key per team, one quota per model, and a 429 before the model is ever called. The model the team runs itself gets a serverless door too. Cloud Run serves it with no standing fleet, and the cold minute is priced in the open before anyone picks that door. Training runs on a TPU VM or a GPU VM, and the artifact lands in the bucket before it is ever an endpoint. Quotas are numbers the team knows, not surprises at noon. The alert watches p95 latency per model version, not the pool total. The cost has a budget line and an alert at 80 percent of it.
Do this: take the lab and deploy the vLLM YAML to a cluster in your own project, then call it through the ClusterIP Service from a pod inside the cluster. Then write down, in one sentence each, why the Vertex endpoint fits the support box and the GKE deployment fits the second model. If you cannot write the sentence, you have not picked. You have copied.
Uses: A production system where a wrong answer costs real users: a support box at peak, a second model the team fine-tuned itself, a training job that runs every night. A team that ships a new model version every month and needs the pipeline to carry it, not a meeting.
Failure modes: The TPU training job runs for hours and nobody watches the cost, because training has no user waiting to complain. The GKE GPU node pool sits idle overnight and bills for empty chairs. The model version changes and the p95 alert never moves, because the alert watches the endpoint total and the new version is a small slice of it.
Try it: Advanced
The alert watches the endpoint total and a new model version is a small slice of it. What do I watch instead?
Now do it: Deploy the vLLM YAML to a cluster in your own GCP project, then call it through the ClusterIP Service from a pod inside the cluster. Then write one sentence each for why Vertex fits the support box and GKE fits the second model.
At the master level, the AI service is a platform, not a config file. Every new model gets its service from a template: endpoint, secret, topic, workflow, grounding bucket, and alerts, all in version control. A new model fills in five fields and gets the right pipeline. Every model version runs through evals before it serves real users: a fixed set of questions, a fixed set of docs, and a score the team can read. The cost budget is per team, and the quota is raised before the launch, not after the first 429. At 10x the traffic, the prompt cache carries the repeated docs and the endpoints scale behind it. At 100x, the endpoints run in two regions and the workflow picks the nearer one.
Do this: write the five-field template for your team. Model name, grounding bucket, secret name, max output tokens, cost budget per month. Then fill it in for one service you actually run, and check whether any of the five fields is missing today. Fix one. That fix is the platform working.
Uses: A platform team that serves other teams, where a wrong answer is an outage for someone else. An org where the AI pipeline is built once, in a template, and every model inherits it.
Failure modes: The template is right and the services drift, because somebody deployed a model by hand and the template never ran again. The evals pass and the real users fail, because the eval questions never change and the model learned the eval. The cost budget alerts at 80 percent and nobody reads it, because the alert goes to a list nobody owns.
Try it: Master
A new model is about to serve real users. What happens first at this level?
Now do it: Write the five-field template for your team: model name, grounding bucket, secret name, max output tokens, cost budget per month. Fill it in for one service you run and fix one missing field.
Model Garden is the catalog. It holds Google's models and open models, with the size, the context length, and the price on the card. I pick by three numbers. How long is the context the question needs. How fast must the answer come back. How much may one answer cost. A small model with a short prompt answers in 200 ms and costs almost nothing. A large model with a long prompt answers in seconds and costs real money. The name on the card is the last thing I read. The three numbers are the first thing. If two models both fit the job, I pick the smaller one and spend the savings on grounding, because grounding is what makes the answer true.
Training is matrix math at scale, and the chip is the question. A TPU is built for that math. It trains large models faster per dollar than a general GPU, when the training code fits the TPU. A GPU VM is flexible. It trains, it serves, it runs anything, and it is the chip I reach for when the job is small or the code is not written for a TPU. The rule I use: train the big model on a TPU, train the small experiment on a GPU, and never train on the serving endpoint. Training is a batch job. Serving is a request. The TPU VM in the lab is the training door. The endpoint is the serving door. Different doors, different jobs.
Lesson on vLLM serving taught the server itself. This lesson puts it on GKE. The Deployment runs two replicas on a GPU node pool, and the Service is ClusterIP, so only callers inside the cluster can reach it. I serve on GKE when the model is one the team runs itself, when the traffic must never leave the cluster, or when the Vertex endpoint does not carry the model I need. I serve on Vertex when the model is managed, when I want Google to own the scaling, and when the team is small. The two doors do not compete. The Vertex endpoint is the managed door. GKE is the door I own. The question is always about the model and the team, never about the brand.
Lesson 114 verified this against the framework pages, so I do not re-litigate it here. Google's current Well-Architected Framework names six pillars: operational excellence, security, reliability, cost optimization, performance optimization, and sustainability. Sustainability is the sixth. The older Architecture Framework listed system design as one of its six pillars instead, and sustainability was not on it. Both lists are real Google pages. One of them is old. The current page wins. In this lesson, sustainability is the idle GPU node pool and the TPU left running overnight. A fleet sized for the worst day burns power on the quiet days, and right-sizing is the fix.
The endpoint answers 429 when the project passes its allowed rate. The quota is per project and per region, and the default is small. Sixty requests a minute is the number in this lesson. Two requests a second is enough to hit it. The fix is a quota raise, and the raise is a request in the console, not a deploy. I check the quota before every launch, the way I check the disk before every deploy. A launch that needs 500 requests a minute on a 60 quota fails at noon, in front of users, and the error message is the only warning anyone gets. Ask for the raise a week before the launch. Quota raises take time, and launches do not wait.
At 10x the traffic, the prompt cache carries the repeated docs, so the same grounding text does not travel into the model fresh on every request. The endpoint scales its replicas, and the queue absorbs the peaks. The cost budget gets a real alert, because at 10x the token bill is the first thing that surprises a team. At 100x, the design changes. The endpoints run in two regions and the workflow picks the nearer one. The evals run on every model version before it serves, because at that scale a bad version reaches a visible slice of users in minutes. The pipeline does not change. The workflow still reads the secret, the docs, the model, and the answer. The parts around it change, and that is the point. The pipeline is the front door. The model behind it is the system.
The pipeline in this lesson has a front door already. It is the function URL at hop 2. Anyone who finds that URL can ask questions, and every question they ask spends the same project quota. The endpoint allows 60 requests a minute for the whole project. One caller can eat all 60 before noon, and the support team gets the 429s. The fix is a door that names its callers before the model is ever called. That door is API Gateway, and it sits in front of the endpoint, not in front of the function. The function stays the front door for the pipeline. The gateway is the front door for the model.
The shape of a gateway is already taught in Lesson 35. One key per caller, one route per model, one meter that answers 429 before the model is called. That lesson also holds the full GCP file, gcp_openapi.yaml: two paths, /chat-small and /chat-big, each behind its own Vertex AI backend, a key required on every call, and a quota of 100 chat requests a minute. I do not restate that file here. This section is the GCP wiring around it, and the two places where the GCP meter is honest about what it cannot do.
Each team gets its own API key. The gateway checks the key on every call. A call with no key gets a 401 from the gateway, and the model never hears about it. A call with a key gets through to the backend, and the gateway calls the Vertex AI endpoint with the service account attached to the gateway. The caller never holds the credential that calls the model. That is the whole point of the door. When a key leaks, I delete that one key. The model secret in Secret Manager never rotates, the other teams never notice, and the leak costs one revocation instead of one outage.
The key is identity, not a budget. That distinction matters on GCP, because the quota in the OpenAPI file is one shared pool. The 100 requests a minute belong to every key at once. One greedy team can eat the whole pool while the other teams hold valid keys. The key tells me who spent the quota. It does not stop one team from spending all of it. Per-key budgets are the Apigee job below, or a counter in the backend that reads the token counts the model reports.
The file in Lesson 35 counts every call as 1 on one metric, chat-requests. Both paths share it. The small model and the big model draw from the same pool, and the big model costs more per call in tokens. I split the meter when the models stop costing the same. One metric per path, one limit per metric. The fragment below is the delta against the Lesson 35 file. It replaces the x-google-api-management block, nothing else.
x-google-api-management:
metrics:
chat-requests-small:
displayName: Chat requests, small model
valueType: INT64
metricKind: DELTA
chat-requests-big:
displayName: Chat requests, big model
valueType: INT64
metricKind: DELTA
quota:
limits:
chat-requests-small-limit:
metric: chat-requests-small
values: 100
chat-requests-big-limit:
metric: chat-requests-big
values: 20
Prove it parses, with no cloud account:
python3 -c "import yaml; d = yaml.safe_load(open('gcp_gateway_quota.yaml')); print('quota YAML parses:', sorted(d['x-google-api-management']['metrics']))"
Expected output, exactly as it ran in this build:
quota YAML parses: ['chat-requests-big', 'chat-requests-small']
Then each path charges its own metric. /chat-small carries x-google-quota: {chat-requests-small: 1} and /chat-big carries x-google-quota: {chat-requests-big: 1}. The big model gets 20 calls a minute because its answers cost more. The small model keeps 100. A new model version is a new path, not an edit to the old one. /chat-big-v2 gets its own metric and its own limit, and the old path keeps serving until I retire it. The route is the versioning. I never change what /chat-big points at under a live caller.
Here is the honest gap in the GCP meter, and Lesson 35 names it too. The quota counts requests. A short question costs 1. A long question costs 1. The bill counts tokens, and the long question costs 15 times the short one. The gateway cannot see that difference, so the quota cannot enforce it. The cost per call has to come from the token counts the model returns in usageMetadata, read per key, after the call. The gateway logs the key and the route. The backend logs the tokens. The bill per team is the two logs joined, not a number the gateway hands me.
Save this as gcp_gateway_cost.py and run it with python3 gcp_gateway_cost.py. It works the cost per call from the mock token counts in step 2, shows the request meter treating a short and a long question the same, and runs a busy minute through both quotas. No cloud account, no network.
"""gcp_gateway_cost.py -- what one call through the gateway costs, and
what the gateway's quota does to a busy minute.
Uses the token counts the mock in step 2 returns (812 in, 96 out) and
the example token prices from this lesson ($1.25 per million input,
$5.00 per million output). Those are example prices, not a quote.
The gateway quota is the one from Lesson 35's gcp_openapi.yaml:
100 chat requests a minute, shared across every key. The endpoint
quota behind it is the 60 requests a minute from this lesson.
Run: python3 gcp_gateway_cost.py
No cloud account, no network. Plain arithmetic from the lesson's numbers.
"""
INPUT_PRICE = 1.25 # dollars per million input tokens, example price
OUTPUT_PRICE = 5.00 # dollars per million output tokens, example price
GATEWAY_QUOTA_RPM = 100 # chat requests a minute, shared across keys
ENDPOINT_QUOTA_RPM = 60 # requests a minute the endpoint allows
def call_cost(in_tokens, out_tokens):
return in_tokens / 1e6 * INPUT_PRICE + out_tokens / 1e6 * OUTPUT_PRICE
print("one call through the gateway (812 in, 96 out, from the mock):")
print(" cost per call $%.4f, per 1,000 calls $%.2f"
% (call_cost(812, 96), call_cost(812, 96) * 1000))
print()
print("the meter counts requests, not tokens:")
for name, in_tokens in [("short question", 300), ("long question", 4500)]:
print(" %s: %d input tokens, quota cost 1, token cost $%.4f"
% (name, in_tokens, call_cost(in_tokens, 0)))
print(" the long question costs 15 times the tokens and the same 1 quota unit.")
print()
print("a busy minute: 2 requests a second is 120 a minute.")
print(" gateway quota %d/min lets 100 through and answers 429 to 20."
% GATEWAY_QUOTA_RPM)
print(" the endpoint quota %d/min lets 60 of those 100 through"
% ENDPOINT_QUOTA_RPM)
print(" and answers 429 to the other 40.")
print(" 60 answers, 60 429s. 20 of the 429s come from the gateway,")
print(" 40 from the endpoint. A gateway quota above the endpoint")
print(" quota protects nothing. Set the gateway quota at or below")
print(" the endpoint quota, or the endpoint still answers 429.")
Expected output, exactly as it ran in this build:
one call through the gateway (812 in, 96 out, from the mock): cost per call $0.0015, per 1,000 calls $1.50 the meter counts requests, not tokens: short question: 300 input tokens, quota cost 1, token cost $0.0004 long question: 4500 input tokens, quota cost 1, token cost $0.0056 the long question costs 15 times the tokens and the same 1 quota unit. a busy minute: 2 requests a second is 120 a minute. gateway quota 100/min lets 100 through and answers 429 to 20. the endpoint quota 60/min lets 60 of those 100 through and answers 429 to the other 40. 60 answers, 60 429s. 20 of the 429s come from the gateway, 40 from the endpoint. A gateway quota above the endpoint quota protects nothing. Set the gateway quota at or below the endpoint quota, or the endpoint still answers 429.
Read the last block twice. The gateway quota is higher than the endpoint quota, so the gateway lets 100 calls through and the endpoint still answers 429 to 40 of them. The gateway did its job. It just did it at the wrong number. A gateway quota only protects the endpoint when it sits at or below the endpoint quota. I set the gateway quota first, then I read the endpoint quota, then I lower the gateway number until it is the smaller one.
API Gateway is the simple door. One OpenAPI file holds the routes, the keys, and the quotas. It fits one team serving one or two models, where a shared request quota is enough and the token bill is read from the logs after the fact. Apigee is the fuller enterprise option. It meters real tokens per key, so a long question costs more quota than a short one. It carries spike arrests, per-developer products, monetization, and a policy chain that goes past quota into transforms and mediation. The pick rule is one sentence. If the quota has to count tokens per team, or more than a few teams share the door and each needs its own budget, I pay for Apigee. If the door serves one team and the token bill is small enough to read from the logs, API Gateway is enough and I stop there. The mistake is buying Apigee for a door that serves one team, and the other mistake is running API Gateway for ten teams and finding out at the end of the month that one team spent the shared pool.
This step needs your own cloud account, and I cannot run it here. There is no cloud account in this build. Everything below is for your account, and the outputs shown are the shape of what your own account returns, not output from this machine. Replace PROJECT_ID with your own project. Start from the gcp_openapi.yaml file in Lesson 35, with the per-model quota fragment above in place of its x-google-api-management block.
gcloud services enable apigateway.googleapis.com apikeys.googleapis.com --project=PROJECT_ID gcloud api-gateway apis create answer-gateway --project=PROJECT_ID gcloud api-gateway api-configs create answer-config --api=answer-gateway --openapi-spec=gcp_openapi.yaml --project=PROJECT_ID gcloud api-gateway gateways create answer-gw --api=answer-gateway --api-config=answer-config --location=us-central1 --project=PROJECT_ID gcloud services api-keys create --display-name=team-alpha-key --project=PROJECT_ID
The gateway create call returns the gateway URL. The caller adds the key to it as ?key=KEY, because the security scheme in the file is an API key in the query. A call with no key gets a 401 from the gateway. A call with a key reaches the endpoint. When you are done, delete the gateway, the API config, the API, and the key. A gateway left running is a door left open, and a key left lying around is a key somebody else will find.
A 429 from the gateway means the shared pool is gone, and the first question I ask is which key spent it, because the key is the only identity in the log. A 429 from the endpoint after a 200 from the gateway means the gateway quota sits above the endpoint quota, and the fix is a smaller gateway number, not more replicas. A 401 from the gateway means the key is missing or deleted, and the model secret is never the suspect. The gateway is a new hop that can fail on its own, and it adds a few milliseconds to every call. I keep it thin. It checks the key, it counts the call, it routes. It never runs business logic, and it never calls the model twice.
The lesson now has two doors for the model. The Vertex AI endpoint serves the managed model. The GKE deployment serves the model the team runs itself, from a standing fleet. This section is the third door. Cloud Run serves the same self-run model with no standing fleet at all. A request arrives, a GPU spins up, the model loads, the answer comes back, the GPU goes away. The image is the same vLLM image from Lesson 181, and the deploy command starts there. That lesson holds the full section: the EKS, GKE, and AKS node pools, the Deployment YAML, the validation script, and the cold-start minute. I do not restate it. This section is what that lesson never priced.
Lesson 181 deploys the image with one L4 GPU, 4 vCPU, 16 GiB, min-instances 0, max-instances 4, in us-central1. The command below is that command, with one flag added. Concurrency 1 means one request per instance. One GPU serves one request at a time, and a second request at the same moment gets its own instance instead of queueing behind the first one inside a container I cannot see into.
gcloud run deploy support-assistant \ --image vllm/vllm-openai:latest \ --gpu 1 \ --gpu-type nvidia-l4 \ --cpu 4 \ --memory 16Gi \ --concurrency 1 \ --min-instances 0 \ --max-instances 4 \ --region us-central1
This command needs your own cloud account. I cannot run it here. There is no cloud account in this build. The output shown is the shape of what your own account returns, not output from this machine. The default request timeout is five minutes, and a 60 second load fits inside it with room. I raise it only if the first token after the load is slow.
Cloud Run bills a GPU instance for its whole lifetime, not just the half second it spends answering. GPU instances run with CPU always allocated, so the meter runs from the moment the instance starts until it scales to zero. That is the first thing people get wrong about this door. They price the answer and forget the lifetime. A warm request costs half a second of GPU time. A lone question at 3 AM costs the 60 second load plus every minute the instance stays alive after it. The script below prices both, and the break-even for keeping one GPU warm all day.
Save this as gcp_run_math.py and run it with python3 gcp_run_math.py. The rates are example rates for this lesson, not a quote. Read the real ones off the pricing page in your project and put them in the four constants at the top. No cloud account, no network.
"""gcp_run_math.py -- what one vLLM request on Cloud Run costs, and
what keeping one GPU warm costs.
Cloud Run bills a GPU instance for its whole lifetime, not just the
half second it spends answering. GPU instances need CPU always
allocated, so the meter runs from the moment the instance starts
until it scales to zero. The warm-request number below is the floor.
The lifetime numbers are what the bill looks like.
Rates are example rates for this lesson, not a quote. Read the real
ones off the pricing page in your project and put them in the four
constants at the top. The cold-start minute is from Lesson 181.
The answer time is the 420 ms endpoint time from this lesson.
Run: python3 gcp_run_math.py
No cloud account, no network. Plain arithmetic from the lesson's numbers.
"""
GPU_S = 0.0002 # dollars per second for one L4 GPU, example rate
VCPU_S = 0.000025 # dollars per vCPU-second, example rate
MEM_S = 0.0000025 # dollars per GiB-second, example rate
CPUS = 4
MEM_GIB = 16
COLD_LOAD_S = 60 # seconds to load the model, from Lesson 181
ANSWER_S = 0.5 # seconds to answer once warm, 420 ms rounded up
HOLD_MIN = 15 # minutes one lone instance stays alive after a question
RATE_S = GPU_S + CPUS * VCPU_S + MEM_GIB * MEM_S
def cost(seconds):
return seconds * RATE_S
print("one instance-second: 1 L4 GPU + 4 vCPU + 16 GiB = $%.5f/s" % RATE_S)
print()
print("a warm request, the floor (%.1f s of GPU time):" % ANSWER_S)
print(" cost per request $%.5f, per 1,000 warm requests $%.2f"
% (cost(ANSWER_S), cost(ANSWER_S) * 1000))
print()
print("a cold request (%.0f s load + %.1f s answer):" % (COLD_LOAD_S, ANSWER_S))
print(" cost per request $%.5f, about %d times the warm floor"
% (cost(COLD_LOAD_S + ANSWER_S),
round(cost(COLD_LOAD_S + ANSWER_S) / cost(ANSWER_S))))
print()
print("a lone question at 3 AM, instance alive %d minutes after it:"
% HOLD_MIN)
print(" %d s billed, cost $%.2f for one question"
% (COLD_LOAD_S + HOLD_MIN * 60,
cost(COLD_LOAD_S + HOLD_MIN * 60)))
print()
print("keeping one GPU warm:")
print(" business hours, 12 h a day: $%.2f a day, $%.2f a month (30 days)"
% (cost(12 * 3600), cost(12 * 3600) * 30))
print(" all day, min-instances 1: $%.2f a day, $%.2f a month (30 days)"
% (cost(24 * 3600), cost(24 * 3600) * 30))
print()
print("break-even for min-instances 1 all day:")
gaps = cost(24 * 3600) / cost(COLD_LOAD_S + ANSWER_S)
print(" min-instances 1 costs $%.2f a day. One cold load costs $%.5f."
% (cost(24 * 3600), cost(COLD_LOAD_S + ANSWER_S)))
print(" it pays only past %.0f idle gaps a day. The support box in this"
% gaps)
print(" lesson has a handful. Min-instances 0 wins, and the cold minute")
print(" is paid a few times a day, not all day.")
Expected output, exactly as it ran in this build:
one instance-second: 1 L4 GPU + 4 vCPU + 16 GiB = $0.00034/s a warm request, the floor (0.5 s of GPU time): cost per request $0.00017, per 1,000 warm requests $0.17 a cold request (60 s load + 0.5 s answer): cost per request $0.02057, about 121 times the warm floor a lone question at 3 AM, instance alive 15 minutes after it: 960 s billed, cost $0.33 for one question keeping one GPU warm: business hours, 12 h a day: $14.69 a day, $440.64 a month (30 days) all day, min-instances 1: $29.38 a day, $881.28 a month (30 days) break-even for min-instances 1 all day: min-instances 1 costs $29.38 a day. One cold load costs $0.02057. it pays only past 1428 idle gaps a day. The support box in this lesson has a handful. Min-instances 0 wins, and the cold minute is paid a few times a day, not all day.
Read the break-even block twice. Keeping one GPU warm all day costs $29.38, and one cold load costs $0.02057. The warm instance only pays when the traffic goes idle and comes back more than 1,428 times a day. The support box in this lesson goes idle a handful of times a day. Min-instances 0 wins for it, and the cold minute is paid a few times, not all day. The same math flips the moment the traffic is steady. A service with a question every minute never goes idle, so it never pays a cold load, and min-instances 1 is the wrong answer for a different reason. The fleet is warm because the traffic keeps it warm, not because I paid for it.
Two dials, and they pull against each other. Min-instances buys warmth with money. Concurrency buys throughput with latency. I tune them against the cold minute from Lesson 181, which is about 60 seconds for a big model to load.
The tuning drill is one week of real traffic. Start at min-instances 0, concurrency 1, max-instances 4. Count the cold starts and read the p95. If the cold starts are the complaint, raise min-instances to 1 in business hours first. If the p95 climbs under load, lower the concurrency, because the GPU is queueing inside the container. If requests wait at the front door, raise max-instances, and read the cold-start count again, because every new instance is a new model load. Change one dial, watch a day, then change the next. Two dials at once tells me nothing.
Three doors, and the question is always the same two parts. Who owns the model, and what does the traffic look like when nobody is watching. The model is the first part. The traffic shape is the second.
The support box in this lesson is quiet overnight and spiky at lunch, and the model is managed, so the Vertex AI endpoint is the door it uses today. The second model, the one the team runs itself, is quiet too, so it goes to Cloud Run before it goes to GKE. GKE is the door for the day the second model gets steady traffic and the team wants the fleet. The doors do not compete. The traffic picks the door, and the traffic changes, so I re-read the table every quarter and move the model when the shape moves.
A request that waits 60 seconds and then answers is a cold start, not a slow model, and the fix is min-instances or a warmer traffic shape, not a bigger GPU. A request that waits and then gets a 429 from Cloud Run means max-instances is the ceiling, and the fix is a higher cap or a lower concurrency, not more model. An instance that never becomes ready means the model did not fit in 16 GiB of memory, and the fix is a bigger memory flag or a smaller model, not a retry. I read the Cloud Run logs for the container start time first. The start time is the cold minute, written down, and every other number in the incident hangs off it.
This one stayed with me because everything looked right. The secret was right. The endpoint was healthy. The users got nothing, and the dashboards were green.
It was a Friday afternoon. The team rotated the model key in Secret Manager, the way the runbook says: create the new version, test it, disable the old one. The new key worked. The old key was disabled. And every answer failed for two hours. The function read the secret once, at cold start, and kept the key in memory. The running instances never asked Secret Manager again. They held the old key, the old key was disabled, and the model answered every call with a 401. The secret was right. The function was reading a copy from memory.
Three things, and I still do all three. First, the function fetches the secret on every execution, or caches it with a five-minute TTL and no longer. A secret cached forever is a secret that cannot rotate. Second, I added an alert on the 4xx rate from the function, not just on latency. A fast 401 is still a failure, and latency alone will never catch it. Third, the rotation runbook changed: after a rotation, I restart the function instances and watch one real answer come back before I call the rotation done. A rotation is not done when the secret is new. It is done when a user gets an answer with the new secret.
Design the doc-answer service from this lesson for 200,000 users. It answers 5,000 questions a day, with a peak of 500 questions a minute. Every answer must quote the docs it came from. The bill must stay under $300 a month at the expected load, at the example prices in this lesson. The service must survive one region failing.
Shape first. One function takes the question, one topic holds it, one workflow orders the steps: secret, docs, model, write. The model is a Vertex AI endpoint with a minimum replica count of 1, so the first user of the morning never pays the cold start. The docs live in a Cloud Storage bucket in two regions, and the answers are written back under the question ID. The GKE vLLM deployment is the second door, for the model the team runs itself, and it stays behind a ClusterIP Service.
Then the numbers. 5,000 answers a day at 800 tokens in and 200 out is $0.002 an answer, which is $10 a day and $300 a month at the example prices. That is the whole budget, so the prompt is the budget: cap the docs at two per question and the output at 200 tokens, and the numbers hold. The peak is 500 questions a minute, and the default quota is 60 a minute, so the quota raise is the first request I file, to 600 a minute, a week before launch. Little's Law for the sizing: at peak, 500 a minute is about 8.3 a second, each spending 0.507 seconds in the system, so about 5 answers are in flight at once. Concurrency is not the problem. Quota is.
Then the failures. A denied secret stops the pipeline at hop 3 and the user gets a 403 with a question ID that will never have an answer. The alert on the function 4xx rate catches it in minutes, because the incident in this lesson taught me that latency alone never catches a fast 401. A cold endpoint is priced at 2,400 ms, and the minimum replica count removes it. One region failing moves the workflow to the second region, where the bucket copy and the endpoint already stand.
Trade-offs to name. The minimum replica count costs money every hour, even at 3 AM when nobody asks anything. Two regions cost double the endpoints and double the bucket. Capping the docs at two per question keeps the bill under $300, and it also means a question that needs three docs gets a shorter answer. Every one of those costs is visible, and every one is smaller than the cost of an ungrounded answer or a quota failure at noon.
One page with a table: one row per hop, naming the service, the config that matters, and the failure that stops it. Plus the cost math, worked out in the open, the quota raise request number, and one sentence per service saying why it fits. If a row cannot name its failure, the design is not done.
Saved in this browser, so the Lessons index can show your progress across the course.