I have shipped a chatbot the simple way and the right way, and the difference showed up at 2 AM. The simple way was one call to a model with the key pasted in the code. It worked until the key leaked, until the bill arrived, until the model answered a question our docs never said. The right way is this lesson. A chat API that reads its key from Key Vault, an embedding model on AKS that turns the question into a vector, AI Search that finds the real help article, Azure OpenAI that answers from that article, and Azure Monitor watching the whole trip. One request, nine hops, 548 ms. And a second story where the index goes stale and the model answers from memory instead. That is the failure this lesson is about.
Difficulty: Advanced · Know this first: Lesson 111, Lesson 112, Lesson 114, Lesson 218, Lesson 16, Lesson 87
A fresher on my team built a support chatbot the simple way. One call to Azure OpenAI, the key pasted in the code, no grounding. It worked. It answered every question, including ones our help articles never covered. It made up a refund policy that did not exist. A customer quoted it back to us, and we had to tell the customer our own chatbot was wrong. That is the day the simple way died.
We rebuilt it the way this lesson teaches. A chat API takes the question. It reads the key from Key Vault, never from the code. A small embedding model on AKS turns the question into a vector. AI Search finds the real help article. Azure OpenAI answers from that article and names it. Azure Monitor watches the grounding rate, and a Function files the conversation away. Same chatbot. Same model. Different trip. The difference is retrieval, and retrieval is this lesson.
Then the indexer stopped, and the new chatbot failed the same way the old one did. The model answered from memory, because the search returned nothing. The dashboards stayed green, because every response was 200 OK. That failure is the third scenario in this lesson, and it is the one I want you to remember. A chatbot is only as honest as its index.
By the end of this lesson you will have written, in Terraform, the six Azure resources this trip needs: an Azure OpenAI resource, an AI Search service, a storage account for the articles, a Key Vault for the key, and a Log Analytics workspace for the metrics. You will prove the config is valid with no cloud account. You will run the whole RAG shape on your own machine in Python, with real retrieval scores. And you will know how to answer the three questions that decide every Azure AI system: where does the key live, where does the answer come from, and what do you watch to know it is still true.
Every hop charges rent in milliseconds. The bar below is a worked example for one support question, not a measurement from a live system, but the shape is honest: on the grounded path, the model does the real work and everything else is retrieval and plumbing. The model is the biggest slice by far, and it is the one you cannot tune at the API. Step through the lesson or switch the scenario and watch the bar change. The cached path is faster because three hops are skipped. The stale path costs more than the normal one, because the model takes longer to invent an answer than to write one from a document.
Same question, two ways to answer it. The left panel is the managed way: Azure OpenAI runs the model, you pay per token, and the quota is tokens per minute. The right panel is the self-hosted way: a model on AKS, on an NC-series GPU node, served by vLLM, and you pay per node-hour. The left one is this lesson's hop 6. The right one is this lesson's hop 4, and the vLLM serving lesson in full. Read the two panels left to right, then bottom to top: the bottom line is the one that matters.
The two panels answer the same question with opposite bills. The managed model bills per token and stops you at a quota. The hosted model bills per node-hour and stops you at a queue. When the model is big and the traffic is spiky, I take the managed one. When the model is small and the traffic is steady, like the embedding model in this lesson, I take the hosted one. Most real systems use both, and this lesson is one of them.
The budget bar above shows three fixed stories. This simulator lets you write your own. It answers one question: how much traffic can the Azure OpenAI deployment take before the token quota runs out. You set how many questions arrive per minute, how many tokens each one costs, how many answers come from the cache, and how many embedding replicas run on AKS. Move a slider and watch the tokens per minute, the quota use, the embedding load, and the verdict change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: the token quota is a wall, and the cache is the only way past it that costs nothing.
How many questions arrive every minute. 120 is a quiet support chat. 3,000 is a peak. More arrivals means more tokens, and the token quota does not move.
How many tokens one question costs, prompt plus answer. 200 is a short question with one chunk. 4,000 is a long history with five chunks. Longer prompts cost more tokens and more model time.
The share of questions answered from the stored answers, at zero tokens. 30 percent is a normal support chat: the same questions come back every day. 90 percent is a FAQ page with a chat box on it.
How many vLLM embedding replicas run on the AKS GPU nodes. One replica embeds 25 questions a second. More replicas means more embedding capacity, and it does nothing for the token quota. Two different limits, two different fixes.
Try this: (1) Push questions per minute to 3,000 at the defaults. 2,100 requests a minute are uncached, 2,520,000 tokens a minute, 1,050 percent of the 240,000 quota, model serve time jumps to 1,220 ms, and the verdict flips to throttled. Same model, same index. Only the traffic changed. (2) Set questions back to 120 and set cached answers to 0. Tokens rise from 100,800 to 144,000 a minute, quota use from 42 to 60 percent, cost from $12.10 to $17.28 an hour. The cache saves $5.18 an hour at this traffic, and the saving grows with the traffic. (3) Push questions to 3,000 again and raise embedding replicas from 1 to 2. Embedding load falls from 140 percent to 70 percent. The model is still throttled at 1,050 percent of quota, because replicas fix the embedding, not the token quota.
Formulas, in plain terms. Uncached requests = questions per minute times one minus the cached share. Tokens per minute = uncached requests times tokens per request. Quota used = tokens per minute divided by 240,000, the deployment quota. Embedding load = uncached requests per second divided by 25 times the replica count. Cost per hour = tokens per minute times 60, divided by 1,000,000, times $2.00, the teaching rate. Model serve time = 120 ms plus 0.25 ms per token, plus 800 ms when the quota is over 100 percent, because throttled requests wait. Moving a control updates the model serve time on hop 6, the budget bar, and the step panel above.
Do this now, in your own terminal. You will write one Terraform file, prove it is valid without any cloud account, and run the whole RAG shape in Python on your own machine. The apply step needs your own Azure account, and it says so plainly where it starts. Nothing before that step does. The config below is the worked setup from this lesson: Azure OpenAI plus AI Search plus Blob Storage plus Key Vault plus Log Analytics, in one resource group.
terraform version az --version
Versions that ran for this lesson: Terraform 1.9.8. Any 1.5 or newer works. The Azure CLI is not installed in this build, so the az commands below are for your own machine: install the Azure CLI, run az login, and the commands work. The terraform steps below all run with no cloud account and no credentials. This is what terraform version printed in this build:
Terraform v1.9.8 on linux_amd64
Save this as main.tf in an empty folder. Every value you do not know yet is a variable at the top. Fill them in from your own account only when you reach the apply step.
terraform {
required_version = ">= 1.5"
required_providers {
azurerm = {
source = "hashicorp/azurerm"
version = "~> 4.0"
}
}
}
provider "azurerm" {
features {}
}
# ---------- shared inputs: fill these in from your own account ----------
variable "location" {
type = string
default = "eastus"
description = "Azure region for the AI resources"
}
variable "resource_group_name" {
type = string
description = "Resource group that holds every AI resource in this lesson"
}
variable "openai_name" {
type = string
description = "Name of the Azure OpenAI resource, e.g. my-openai-01"
}
variable "search_name" {
type = string
description = "Name of the AI Search service, e.g. my-search-01"
}
variable "storage_name" {
type = string
description = "Name of the storage account for training data and artifacts (lowercase letters and numbers only, 3-24 chars)"
}
variable "vault_name" {
type = string
description = "Name of the Key Vault that holds the OpenAI key, e.g. my-vault-01"
}
variable "tenant_id" {
type = string
description = "Your Microsoft Entra tenant ID (az account show --query tenantId)"
}
# ---------- one resource group for everything ----------
resource "azurerm_resource_group" "ai" {
name = var.resource_group_name
location = var.location
}
# ---------- Azure OpenAI: the managed model ----------
resource "azurerm_cognitive_account" "openai" {
name = var.openai_name
location = azurerm_resource_group.ai.location
resource_group_name = azurerm_resource_group.ai.name
kind = "OpenAI"
sku_name = "S0"
}
# ---------- AI Search: the managed vector search ----------
resource "azurerm_search_service" "rag" {
name = var.search_name
resource_group_name = azurerm_resource_group.ai.name
location = azurerm_resource_group.ai.location
sku = "standard"
replica_count = 1
partition_count = 1
}
# ---------- Blob Storage: training data and artifacts ----------
resource "azurerm_storage_account" "data" {
name = var.storage_name
resource_group_name = azurerm_resource_group.ai.name
location = azurerm_resource_group.ai.location
account_tier = "Standard"
account_replication_type = "LRS"
}
# ---------- Key Vault: the OpenAI key lives here, nowhere else ----------
resource "azurerm_key_vault" "secrets" {
name = var.vault_name
location = azurerm_resource_group.ai.location
resource_group_name = azurerm_resource_group.ai.name
tenant_id = var.tenant_id
sku_name = "standard"
soft_delete_retention_days = 7
purge_protection_enabled = false
}
# ---------- Log Analytics: where Azure Monitor sends the model metrics ----------
resource "azurerm_log_analytics_workspace" "monitor" {
name = "${var.resource_group_name}-logs"
location = azurerm_resource_group.ai.location
resource_group_name = azurerm_resource_group.ai.name
sku = "PerGB2018"
retention_in_days = 30
}
terraform fmt terraform init -backend=false terraform validate
Expected output, exactly as it ran in this build. fmt prints nothing when the file is already formatted. That silence is the pass. init downloads the Azure provider, and validate checks the config against it:
Terraform has been successfully initialized! Success! The configuration is valid.
One honest note. The provider download is a few hundred MB. If your /tmp is a small tmpfs, the download can fail with no space left. If you hit that, point the temp dir at a bigger disk and run it again: TMPDIR=$HOME/.tmp terraform init -backend=false. The config was fine. The disk was full. The provider this build downloaded was hashicorp/azurerm v4.81.0.
No cloud account, no network, no API key. Save this as rag_local.py and run it with python3 rag_local.py. It does the three RAG steps in plain Python: embed the question as a word bag, retrieve the best chunk by cosine similarity, and generate an answer from that chunk. In Azure, an embedding model on AKS does step 1, AI Search does step 2, and Azure OpenAI does step 3. The shape is the same. Only the scale changes.
"""rag_local.py — the RAG shape, on your own machine, no cloud account.
RAG is three steps, and this script does all three in plain Python:
1. EMBED the question: here, a word bag. In Azure, an embedding model
on AKS (vLLM) or Azure OpenAI turns the question into a vector.
2. RETRIEVE: score every document chunk against the question and take
the best one. In Azure, AI Search does this over a vector index.
3. GENERATE: hand the best chunk to the model as grounding and let it
answer from the chunk, not from memory.
Run: python3 rag_local.py
No cloud account, no network, no API key. Plain arithmetic + strings.
"""
import math
import re
STOPWORDS = set(
"a an the is are was were do does did how what when where who why "
"i my me to of in on for with and or it its this that can could would "
"please".split()
)
DOCS = {
"password-reset": (
"To reset your password, open the sign-in page, choose "
"Forgot password, and enter your work email. You get a reset link "
"that works for 30 minutes. Use it once, then it stops working."
),
"billing-address": (
"To change the billing address on your account, open Settings, "
"choose Billing, and edit the address. The change applies to the "
"next invoice. Past invoices keep the old address."
),
"cancel-plan": (
"To cancel your plan, open Settings, choose Plan, and choose "
"Cancel. Your plan runs until the end of the paid month, then it "
"stops. You keep your data for 30 days after it stops."
),
}
def words(text):
return [w for w in re.findall(r"[a-z]+", text.lower()) if w not in STOPWORDS]
def embed(text):
vec = {}
for w in words(text):
vec[w] = vec.get(w, 0) + 1
return vec
def cosine(a, b):
dot = sum(a[w] * b.get(w, 0) for w in a)
na = math.sqrt(sum(v * v for v in a.values()))
nb = math.sqrt(sum(v * v for v in b.values()))
if na == 0 or nb == 0:
return 0.0
return dot / (na * nb)
def retrieve(question, docs):
q = embed(question)
scored = sorted(
((cosine(q, embed(text)), name) for name, text in docs.items()),
reverse=True,
)
return scored
def answer(question, docs):
scored = retrieve(question, docs)
if not scored:
return None, scored
best_score, best_name = scored[0]
if best_score == 0:
return None, scored
chunk = docs[best_name]
return (
"Based on our help article '%s': %s" % (best_name, chunk),
scored,
)
if __name__ == "__main__":
QUESTION = "how do I reset my password"
print("question:", QUESTION)
print()
print("retrieval scores (cosine similarity, 0 to 1):")
for score, name in retrieve(QUESTION, DOCS):
print(" %-16s %.3f" % (name, score))
print()
ans, _ = answer(QUESTION, DOCS)
print("grounded answer:")
print(" ", ans)
print()
print("stale-index check: same question, index with zero documents:")
ans2, scored2 = answer(QUESTION, {})
print(" best score:", "none — no chunks retrieved" if ans2 is None else scored2[0])
print(" answer:", ans2 if ans2 else "NO GROUNDING — the model would answer from memory")
Expected output, exactly as it ran in this build:
question: how do I reset my password retrieval scores (cosine similarity, 0 to 1): password-reset 0.516 cancel-plan 0.000 billing-address 0.000 grounded answer: Based on our help article 'password-reset': To reset your password, open the sign-in page, choose Forgot password, and enter your work email. You get a reset link that works for 30 minutes. Use it once, then it stops working. stale-index check: same question, index with zero documents: best score: none — no chunks retrieved answer: NO GROUNDING — the model would answer from memory
Read the scores. The password-reset article scores 0.516. The other two score 0.000, because they share no words with the question. That is retrieval working. Then read the last two lines. With zero documents in the index, the answer is NO GROUNDING. That is the stale-index scenario from this lesson, in six lines of Python. The model would answer from memory, and the customer would get an invented reset link.
Save this as chat_request.json. It is the exact request body the chat API sends to Azure OpenAI: the system message with the grounding rule, and the user message with the question plus the retrieved chunk.
{"max_tokens":200,"messages":[{"content":"You answer only from the documents given. If the documents do not say it, say you do not know.","role":"system"},{"content":"How do I reset my password?\n\nDocuments:\n- password-reset: To reset your password, open the sign-in page, choose Forgot password, and enter your work email. You get a reset link that works for 30 minutes.","role":"user"}],"temperature":0.2}
Then prove it parses, with no cloud account:
python3 -m json.tool chat_request.json
Expected output: the same JSON printed back, formatted, with no error. That is the pass. The two fields to read are temperature and the system message. Temperature 0.2 keeps the answer close to the documents. The system message is the guardrail: answer only from the documents given, and say you do not know when the documents do not say it. Change the system message to "be helpful" and the model will invent. Change the temperature to 1.0 and the answer will wander. Both changes are one word, and both break the grounding.
This step needs your own Azure account, and I cannot run it here. There is no Azure account in this build. Everything below is for your account, and the outputs shown are the shape of what your own account returns, not output from this machine. Do not paste them into a report as if they ran here.
First the values. Save them as terraform.tfvars. Use your real names, not these placeholders:
location = "eastus" resource_group_name = "rg-ai-support" openai_name = "my-openai-01" search_name = "my-search-01" storage_name = "myaistore01" vault_name = "my-vault-01" tenant_id = "00000000-0000-0000-0000-000000000000"
Get your tenant ID from az account show --query tenantId after az login. The storage account name must be lowercase letters and numbers only, 3 to 24 characters, and it must be unique across all of Azure. Then run:
terraform plan terraform apply
Read the plan before you type yes. It should create one resource group, one Azure OpenAI resource, one AI Search service, one storage account, one Key Vault, and one Log Analytics workspace. Nothing else. Then create a model deployment inside the OpenAI resource, from your account:
az cognitiveservices account deployment create --name my-openai-01 --resource-group rg-ai-support --deployment-name chat --model-name gpt-4o-mini --model-version 2024-07-18 --model-format OpenAI --sku-capacity 10 --sku-name Standard
Then ask it the question from this lesson, with the key read from Key Vault, not from a file:
KEY=$(az keyvault secret show --vault-name my-vault-01 --name openai-key --query value -o tsv) curl https://my-openai-01.openai.azure.com/openai/deployments/chat/chat/completions?api-version=2024-10-21 -H "Content-Type: application/json" -H "api-key: $KEY" -d @chat_request.json
Expected output, the shape of it: a JSON response with a choices list, one entry, and a message.content that gives the reset steps from your article. If the content invents a policy, read the system message in chat_request.json first. The model is doing what it was told. When you are done, tear it down: terraform destroy. An OpenAI resource and a search service left running bill by the hour.
Change one thing and run it again. In rag_local.py, change the question to "how do I cancel my plan" and run it. The cancel-plan article should score highest and the answer should come from it. Then delete all three documents from DOCS and run it again. The answer is NO GROUNDING. The retrieval is the difference between an answer from a document and an answer from memory, and you just watched it flip.
Three questions. No score is saved anywhere — this is just you against the material.
AI on Azure is not one thing. It is four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how many models you own. They are about how much of the answer is deliberate: where the key lives, where the answer comes from, and what you watch to know it is still true. Read them in order. Each level fixes the failure that hurt the most at the level below it.
At the basic level, the chatbot is one call to Azure OpenAI. The key is pasted in the code. There is no retrieval, no search, no grounding. The model answers from memory, and it answers everything, including questions the docs never covered. It works. The customer gets an answer. That is the whole system, and it is how every Azure AI project starts.
Do this: run the lab in this lesson, then ask the model a question your documents never answer, with no chunks in the request. Watch it answer anyway, confidently. That is the failure the next level fixes, and you have just seen it with your own eyes.
Uses: A demo for one stakeholder. A first version built to answer one question: does the model answer at all. I use this level to learn, never to serve real customers.
Failure modes: The key leaks, because it is in the code and the code is in git. The model invents a policy, because nothing grounds it. The bill arrives, because nobody counted the tokens.
Try it: Basic
At the basic level, where does the key live and what does the model read before it answers?
Now do it: Run the lab in this lesson in your own Azure account, then ask the model a question your documents never answer with no chunks in the request and watch it answer anyway.
At the intermediate level, the chatbot is grounded. The help articles live in Blob Storage. AI Search indexes them. The API embeds the question, retrieves the top chunks, and sends them with the question to Azure OpenAI. The key lives in Key Vault and the app reads it at runtime with its managed identity. The system message says, in plain words, answer only from the documents. This is the level this lesson teaches in full.
Do this now: run the lab in this lesson, then change the system message in chat_request.json from "answer only from the documents given" to "be helpful" and ask the password question again. Watch the answer wander. Then change it back. You just proved the system message is a guardrail, not decoration.
Uses: A small team serving a real support chat. A help centre where a wrong answer is an embarrassment, not an outage. Any system where the callers are customers you can apologise to when something breaks.
Failure modes: The indexer stops and nobody notices, so the answers drift away from the documents. The token bill grows, because nobody watches tokens per minute. The cache is missing, so every repeated question pays full price in tokens.
Try it: Intermediate
I change the system message from answer only from the documents given to be helpful. What happens?
Now do it: Run the lab, then change the system message in chat_request.json from answer only from the documents given to be helpful and ask the password question again, then change it back. Do this in your own Azure account.
At the advanced level, the team owns the whole pipeline. The embedding model runs on AKS on NC-series GPUs, served by vLLM, so the embedding is not a second managed bill. A Function re-indexes the articles when they change, and stores every conversation for review. Azure Monitor watches the grounding rate, the tokens per minute against the quota, and the cache hit rate. A cached answer serves the repeated questions at zero tokens. The TPM quota has an alert at 70 percent, not at 100, because at 100 the requests are already failing.
Do this: take the lab and add the cache: store the answer from the grounded trip in Blob Storage under the question, and serve it on the next identical question. Then ask the same question twice and read the token count on the second one. Zero. If you cannot see the zero, the cache is not wired in.
Uses: A production support chat where a wrong answer costs a customer: a shop at peak, a billing question that must never be answered from memory, a team that ships new help articles every week and needs the index to keep up.
Failure modes: The AKS embedding pods restart and the model goes cold, so retrieval stalls while the model loads. The cache serves a stale answer, because the articles changed and the cache did not. The Functions pipeline fails quietly, because it runs after the answer and nobody watches it.
At the advanced level, one more hop belongs in front of the model. The chat API calls Azure OpenAI straight today, with a key from Key Vault. I put Azure API Management, APIM, in front of it. The client calls APIM first. APIM checks the caller, counts the tokens, picks the deployment, then calls Azure OpenAI. The chat API still does the retrieval work. APIM does the door work. The generic gateway shape lives in Lesson 35: one door, one key per team, tokens counted at the edge, the model picked by a rule, and a runnable ai_gateway.py that models all three. I do not repeat that teaching here. This part is the Azure wiring only.
Subscription keys. Each team gets a subscription key from APIM. That key is not the OpenAI key. The caller sends it in the Ocp-Apim-Subscription-Key header. APIM checks it, then calls Azure OpenAI with its own managed identity. The OpenAI key never leaves Azure, and no caller ever sees it. When a team key leaks, I revoke that one subscription in APIM. The OpenAI key never changes, and the other teams never notice. That is the whole point of the door: a leaked caller key is a small fix, not a rotation for everyone.
Per-model products. I make one product per deployment. A chat product routes to the chat deployment. An embeddings product routes to the embedding deployment. A team subscribes to the products it needs, and each subscription is a separate key. A team with only the chat product cannot spend tokens on the embedding deployment, because it holds no key for it. When a new model deployment lands, I add a product for it. The callers learn one new key, and the old keys keep working.
Two limits per key, and they bind at different times. rate-limit-by-key caps calls: 100 calls a minute per subscription key. azure-openai-token-limit caps tokens: 60,000 tokens a minute per key. At 1,200 tokens a request, the token limit binds first. 60,000 divided by 1,200 is 50 requests a minute, and the call cap never gets reached. At 200 tokens a request, the call limit binds first. The token limit would allow 300 requests a minute, and the call cap stops it at 100. I set both, because a short prompt and a long prompt hit different walls, and one limit alone leaves the other open.
The policy below is the whole door in one file. Save it as apim-policy.xml and attach it to the API in your own APIM instance, on the inbound side. The apply step needs your own Azure account. Everything before it runs on your own machine.
<policies>
<inbound>
<validate-jwt header-name="Authorization" failed-validation-httpcode="401" require-expiration-time="true">
<openid-config url="https://login.microsoftonline.com/00000000-0000-0000-0000-000000000000/v2.0/.well-known/openid-configuration" />
<required-claims>
<claim name="aud" match="any">
<value>api://my-chat-api</value>
</claim>
</required-claims>
</validate-jwt>
<rate-limit-by-key calls="100" renewal-period="60" counter-key="@(context.Subscription.Id)" />
<azure-openai-token-limit tokens-per-minute="60000" counter-key="@(context.Subscription.Id)" estimate-prompt-tokens="true" />
<cache-lookup vary-by-developer="true" vary-by-developer-group="false" downstream-caching-type="none" />
<set-backend-service base-url="https://my-openai-01.openai.azure.com" />
</inbound>
<backend>
<authentication-managed-identity resource="https://cognitiveservices.azure.com" />
<forward-request />
</backend>
<outbound>
<cache-store duration="3600" />
</outbound>
<on-error />
</policies>
Read it top to bottom. validate-jwt runs first. A caller with no token, or a token for another API, gets a 401 from APIM and never reaches the model. Then the two limits run, in order. The call cap counts the request. The token limit counts the tokens, and estimate-prompt-tokens tells APIM to count the prompt before it sends it, so a request over budget is refused at the door and never billed. Then the cache lookup runs, and set-backend-service points the call at the OpenAI resource. In the backend section, authentication-managed-identity tells APIM to prove itself to Azure OpenAI with its own identity. The outbound section stores the answer in the cache for an hour.
JWT validation. The token comes from Microsoft Entra ID. APIM checks the signature against the OpenID config and checks the audience claim. The audience has to be this API. A token issued for another API fails here, even when the signature is good. The failure mode to respect is a wrong audience or a wrong issuer in this policy. It locks out every caller at once, and the error is a 401 from APIM, not from the model. I test the policy with one subscription before I roll it to every product. One key, one call, one 200, then the rest.
Response caching, and the key that makes it safe. APIM can store an answer and serve it again at zero tokens. That is the same saving the simulator shows. The catch is the cache key. The key has to include the question. The default key is the URL plus the headers I name. A POST body is not in it. So I cache only calls where the question rides in a header or a query string I key on. If I cache a POST by URL alone, one question can return another question's answer. I test it with two different questions before I trust it. And a cached billing answer goes stale the same way the Blob cache in this lesson does, so the cache duration stays short and a changed article clears it.
Cost and TPM, with the simulator's numbers. The deployment quota is 240,000 tokens a minute. The teaching rate is $2.00 per 1M tokens. The APIM limit is per key. The Azure quota is per deployment. The sum of the keys has to fit under the quota, or the keys will fight each other at the Azure door. The script below runs the numbers. Save it as apim_tpm_math.py and run it with python3 apim_tpm_math.py. No cloud account, no network, plain arithmetic from this lesson.
"""apim_tpm_math.py: per-key APIM limits against the deployment TPM quota.
Lesson numbers: deployment quota 240,000 tokens/min, $2.00 per 1M tokens,
1,200 tokens per request at the simulator defaults.
Run: python3 apim_tpm_math.py (no cloud account, no network).
"""
QUOTA = 240000
RATE = 2.00
TOKENS_PER_REQ = 1200
def reqs_per_min(token_limit, tokens=TOKENS_PER_REQ):
return token_limit // tokens
def cost_per_hour(tpm):
return tpm * 60 / 1_000_000 * RATE
print("one key, token limit 60,000/min, 1,200 tokens per request:")
print(" %d requests/min before APIM answers 429" % reqs_per_min(60000))
print(" same key, 200-token requests: %d requests/min before the token limit binds"
% reqs_per_min(60000, 200))
print(" the 100 calls/min rate limit binds first there")
print()
print("three keys at 60,000/min each:")
combined = 3 * 60000
print(" combined %d tokens/min, %.0f%% of the 240,000 quota"
% (combined, combined / QUOTA * 100))
print(" cost at full use: $%.2f/hour" % cost_per_hour(combined))
print()
print("three keys at 240,000/min each (the full quota per key):")
combined2 = 3 * 240000
print(" combined %d tokens/min, %.0f%% of quota, so Azure answers 429 to most of the traffic"
% (combined2, combined2 / QUOTA * 100))
print()
print("lesson defaults through APIM, 120 questions/min, 1,200 tokens, 30% cached:")
tpm = 120 * (1 - 0.30) * 1200
print(" %.0f tokens/min, %.0f%% of quota, $%.2f/hour"
% (tpm, tpm / QUOTA * 100, cost_per_hour(tpm)))
Expected output, exactly as it ran in this build:
one key, token limit 60,000/min, 1,200 tokens per request: 50 requests/min before APIM answers 429 same key, 200-token requests: 300 requests/min before the token limit binds the 100 calls/min rate limit binds first there three keys at 60,000/min each: combined 180000 tokens/min, 75% of the 240,000 quota cost at full use: $21.60/hour three keys at 240,000/min each (the full quota per key): combined 720000 tokens/min, 300% of quota, so Azure answers 429 to most of the traffic lesson defaults through APIM, 120 questions/min, 1,200 tokens, 30% cached: 100800 tokens/min, 42% of quota, $12.10/hour
Read the middle lines twice. Three keys at 60,000 tokens a minute each add up to 180,000, which is 75 percent of the 240,000 quota. That leaves headroom, and it costs $21.60 an hour at full use. Three keys at the full 240,000 each add up to 720,000, which is 300 percent of quota. Each key looks fine on its own. Together they push three times the quota, and Azure answers 429 to most of the traffic. The fix is to size the keys from the quota down, not from each team up. The last lines tie back to the simulator. At the lesson defaults, 120 questions a minute at 1,200 tokens with 30 percent cached is 100,800 tokens a minute, 42 percent of quota, and $12.10 an hour. Every answer APIM serves from its cache is tokens the meter never counts.
Do this now, in your own Azure account. Create an APIM instance and one product for the chat deployment. Attach the policy above to the API. Then set the token limit low for one test key, 1,000 tokens a minute, and send three password questions at 1,200 tokens each. The first passes. The second is refused by APIM with a 429. Azure OpenAI never sees it, and the bill never sees it. Then read the APIM analytics for that key. The refused call is there, with the key name on it. That log line is the audit trail. When the bill jumps next month, that line says which key did it.
Failure modes. A key with no token limit pays for someone else's traffic, because one team can burn the whole deployment quota and the other teams get the 429s. Limits that add up past the quota do the same thing more politely, because every key looks right alone. A JWT policy with the wrong audience locks out every caller at once, and the 401 comes from APIM, so check the policy before you check the model. A cache key that misses the question serves one caller another caller's answer, and the response is still 200 OK. The door only works when the keys are sized from the quota down, the token is tested on one key first, and the cache key carries the question.
Try it: Advanced
Each key caps calls at 100 a minute and tokens at 60,000 a minute. At 1,200 tokens a request, which limit binds first?
Now do it: Work the two cases on paper: 60,000 divided by 1,200 and 60,000 divided by 200, then check the numbers against the limits in your own APIM setup. I ran the division here: 50 and 300.
At the master level, the AI system is a platform, not a config file. Every chatbot gets its resources from a template: OpenAI, Search, storage, Key Vault, monitoring, all in version control. A new chatbot fills in five fields and gets the right shape. The grounding rate has an alert and an eval set: a list of questions with known-good answers, run every night, so a stale index or a loose prompt is caught by a test, not by a customer. At 10x the traffic, the TPM quota is raised before the peak, not during it. At 100x, the embedding runs on its own node pool, the cache takes the repeated questions, and the model only sees what is new.
Do this: write the five-field template for your team. Service name, model deployment, search index, Key Vault name, cache on or off. Then fill it in for one chatbot you actually run, and check whether its key is in Key Vault, its index has a freshness check, and its grounding rate has an alert. Fix one. That fix is the platform working.
Uses: A platform team that serves other teams, where a wrong answer is an outage for someone else. An org where the AI shape is decided once, in a template, and every chatbot inherits it.
Failure modes: The template is right and the chatbots drift, because somebody edited a resource by hand and the template never ran again. The eval set goes stale, because the help articles changed and the known-good answers did not. The quota is raised for the peak and never lowered, and the bill becomes the new normal.
Try it: Master
At the master level, how does a stale index get caught before a customer sees it?
Now do it: Write the five-field template for your team: service name, model deployment, search index, Key Vault name, cache on or off. Fill it in for one chatbot you run and fix one missing field.
Azure OpenAI is a managed model. You call it, you pay per token, and Microsoft runs the GPUs. Pick it when the model is big and the traffic is spiky, because the quota scales with the call and you never wait for a node. Azure Machine Learning is a training and hosting platform. You train a model, or fine-tune one, and deploy it to an endpoint you own. Pick it when the model is yours: a fine-tuned model on your data, a model you trained yourself, or a model that needs a custom serving shape. The two do not compete. Azure OpenAI answers the question. Azure ML builds the model that answers the question. Most teams use Azure OpenAI first and reach for Azure ML when the managed model cannot be taught what they need.
The vector databases lesson taught the shape: vectors in, nearest neighbours out. AI Search is that shape, managed, with a text index beside the vector index and an indexer that reads your blobs for you. Pick AI Search when the documents live in Blob Storage and you want the indexer to do the chunking and embedding for you. Pick a standalone vector database when the vectors come from somewhere else, or when you need a query shape AI Search does not offer. The rule I use: if the indexer saves me a pipeline, I take AI Search. If the pipeline already exists and the vectors are already built, I take the database. Either way, the index is only as fresh as the job that fills it, and the freshness check is not optional.
NC-series VMs are Azure's GPU machines, and AKS is where they live in this lesson. The embedding model runs on them, served by vLLM, the way the vLLM serving lesson teaches it. The model stays loaded on the GPU, requests queue behind it, and the answer is a vector. Two things to know. First, the GPU node pool is separate from the CPU node pool, so the embedding never steals CPU from the API and the API never steals GPU from the model. Second, a cold pod is slow: when a pod restarts, the model loads from disk to GPU memory, and the first questions wait. The fix is a readiness probe that only passes when the model is loaded, and a minimum replica count of one, so there is always a warm model. A queue that grows because the model is cold looks exactly like a queue that grows because the traffic is high. The probe tells them apart.
Tokens per minute is a quota, not a bill. When the tokens you send in a minute pass the deployment quota, Azure answers 429 and the request waits or fails. The cache is the cheapest way past it, because a cached answer costs zero tokens. Write the script below as azure_ai_math.py and run it with python3 azure_ai_math.py. It computes the quota use, the cache saving, the embedding replicas, and the model serve time from the lesson. No cloud account, no network. Plain arithmetic.
"""azure_ai_math.py — the three numbers that decide an Azure AI rollout.
1. TPM quota: Azure OpenAI bills attention in tokens per minute. When
the tokens you send in a minute pass the deployment quota, Azure
answers 429 and the request waits or fails.
2. Cache savings: a cached answer costs zero tokens. The cache hit rate
is the cheapest capacity you will ever buy.
3. Embedding capacity: every uncached question needs one embedding call
on AKS before AI Search can retrieve. One NC-series replica embeds
25 questions a second.
Teaching rates used here: $2.00 per 1M tokens, 240,000 tokens/min quota.
Run: python3 azure_ai_math.py
No cloud account, no network. Plain arithmetic from the lesson's numbers.
"""
def uncached_rpm(rpm, cache_hit_pct):
return rpm * (1 - cache_hit_pct / 100)
def tokens_per_min(rpm, tokens_per_req, cache_hit_pct):
return uncached_rpm(rpm, cache_hit_pct) * tokens_per_req
def quota_pct(tpm, quota=240000):
return tpm / quota * 100
def cost_per_hour(tpm, dollars_per_million=2.00):
return tpm * 60 / 1_000_000 * dollars_per_million
def embed_replicas_needed(uncached, per_replica_qps=25):
import math
return max(1, math.ceil(uncached / 60 / per_replica_qps))
print("TPM quota, 120 requests/min, 1200 tokens each, 30% cached:")
tpm = tokens_per_min(120, 1200, 30)
print(" uncached: %.0f req/min -> %.0f tokens/min, %.0f%% of the 240k quota"
% (uncached_rpm(120, 30), tpm, quota_pct(tpm)))
print(" cost: $%.2f/hour" % cost_per_hour(tpm))
print()
print("same traffic, cache off:")
tpm0 = tokens_per_min(120, 1200, 0)
print(" uncached: %.0f req/min -> %.0f tokens/min, %.0f%% of quota"
% (uncached_rpm(120, 0), tpm0, quota_pct(tpm0)))
print(" cost: $%.2f/hour — the cache saves $%.2f/hour"
% (cost_per_hour(tpm0), cost_per_hour(tpm0) - cost_per_hour(tpm)))
print()
print("peak: 3000 requests/min, 1200 tokens each, 30% cached:")
tpm_peak = tokens_per_min(3000, 1200, 30)
print(" %.0f tokens/min, %.0f%% of quota — over 100%% means 429s"
% (tpm_peak, quota_pct(tpm_peak)))
print(" embedding: %.0f uncached req/min needs %d replica(s) at 25 q/s each"
% (uncached_rpm(3000, 30), embed_replicas_needed(uncached_rpm(3000, 30))))
print(" at 120 req/min the same pool needs %d replica(s)"
% embed_replicas_needed(uncached_rpm(120, 30)))
print()
print("model serve time, 120 ms base + 0.25 ms per token + 800 ms when throttled:")
for tokens in (200, 1200, 4000):
print(" %4d tokens -> %4d ms unthrottled, %4d ms throttled"
% (tokens, round(120 + 0.25 * tokens), round(120 + 0.25 * tokens + 800)))
Expected output, exactly as it ran in this build:
TPM quota, 120 requests/min, 1200 tokens each, 30% cached: uncached: 84 req/min -> 100800 tokens/min, 42% of the 240k quota cost: $12.10/hour same traffic, cache off: uncached: 120 req/min -> 144000 tokens/min, 60% of quota cost: $17.28/hour — the cache saves $5.18/hour peak: 3000 requests/min, 1200 tokens each, 30% cached: 2520000 tokens/min, 1050% of quota — over 100% means 429s embedding: 2100 uncached req/min needs 2 replica(s) at 25 q/s each at 120 req/min the same pool needs 1 replica(s) model serve time, 120 ms base + 0.25 ms per token + 800 ms when throttled: 200 tokens -> 170 ms unthrottled, 970 ms throttled 1200 tokens -> 420 ms unthrottled, 1220 ms throttled 4000 tokens -> 1120 ms unthrottled, 1920 ms throttled
Read the peak lines twice. At 3,000 questions a minute, the tokens hit 1,050 percent of quota. That is not a slow day. That is 429s for most of the traffic. The cache at 30 percent does not save the peak, because 70 percent of 3,000 is still 2,100 uncached questions. The fix at the peak is a higher quota, or a higher cache rate, or fewer tokens per request. The embedding only needs 2 replicas at that peak, because embedding is cheap and the model is the wall. Two different limits, two different fixes, and the math tells you which one you are hitting.
Key Vault holds secrets: the OpenAI key, the search key, the storage connection string. App Configuration holds settings that are not secrets: the model deployment name, the index name, the temperature, the cache on or off flag. The split matters. A secret in App Configuration is a leak with a nicer name. A setting in Key Vault is a secret that changes every time someone edits a flag. The app reads the secret from Key Vault at runtime with its managed identity, and reads the settings from App Configuration at startup. When the key rotates, the app picks up the new one from the vault. When the temperature changes, the app picks up the new value from App Configuration. Neither change needs a deploy, and neither value ever sits in the code.
At 10x the traffic, the TPM quota is the first wall, and it is raised before the peak, not during it. The cache rate is pushed up, because every cached question is a question the model never sees. The embedding gets its own node pool, so a slow model never starves the API. The grounding rate gets an alert, because one bad index at 10x is ten times the wrong answers. At 100x, the shape changes. The cache takes the repeated questions. The embedding runs on its own node pool with its own autoscaler. The model only sees what is new, and the prompt is cut to the smallest history and the fewest chunks that still ground the answer. The resources do not change. The numbers around them change, and that is the point. The pipeline is the system. The model is one hop in it.
This one stayed with me because the failure was polite. Nothing crashed. Nothing paged. The chatbot just started making things up, one answer at a time, and a customer found it before engineering did.
It was a Thursday morning. The support chatbot runs the trip in this lesson: API, Key Vault, embedding on AKS, AI Search, Azure OpenAI. Two days earlier, the team had changed the help articles. The indexer was supposed to pick up the change and re-index. It did not. The indexer had failed on a storage permission, and the failure was silent: the indexer logged an error and the schedule kept running, failing the same way every hour. The index went stale. New articles never arrived, and one old article stayed that should have been replaced. The model kept answering, because the API still sent it the question, and the search still returned whatever the stale index held. For questions about the changed articles, the search returned nothing, and the model answered from memory. It invented a reset link that lasts 24 hours. The real one lasts 30 minutes. Customers followed the invented link, and it did not work.
Three things, and I still do all three. First, the indexer got an alert. When an indexer run fails, somebody is paged, the same as any other failed job. A pipeline that fails quietly is a pipeline that will fail for two days. Second, the grounding rate got an alert. When the share of answers with zero retrieved chunks rises past 5 percent in an hour, the team is paged. The status code cannot catch this failure, because the response is 200 OK. The grounding rate can. Third, I added an eval set: a list of questions with known-good answers, run every night against the live chatbot. A stale index or a loose prompt is now caught by a test, not by a customer. The eval runs at 3 AM. The customer never has to find it first.
Design the AI support chatbot for a shop that gets 3,000 questions a minute at peak, with help articles that change twice a week, a billing question that must never be answered from memory, and a budget that caps model spend at $15 an hour. The shop speaks to customers in a chat box. The billing questions are one in five of all questions.
Shape first. The chatbot is the trip in this lesson: chat API, Key Vault for the key, embedding on AKS, AI Search over the help articles in Blob Storage, Azure OpenAI for the answer, Azure Monitor watching the grounding rate, and a Function that re-indexes when the articles change. The billing questions get a hard rule in the system message and in the API: when the search returns zero chunks for a billing question, the API answers "I do not know, let me get a person" instead of calling the model. A billing question is never answered from memory.
Then the numbers. At 3,000 questions a minute with a 30 percent cache rate, 2,100 are uncached. At 1,200 tokens each, that is 2,520,000 tokens a minute, 1,050 percent of the 240,000 quota, and $302.40 an hour at the teaching rate. The $15 an hour budget does not survive the peak. The fixes, in order: raise the cache rate, because the same questions come back every day; cut the tokens per request, with a shorter history and three chunks instead of five; and raise the TPM quota before the peak. At 90 percent cache, the uncached 300 questions cost 360,000 tokens a minute, still 150 percent of quota, so the quota must rise too. The budget is met by the cache and the quota together, not by either one alone.
Then the failures. The indexer gets an alert on failure, and the grounding rate gets an alert past 5 percent ungrounded in an hour. The articles change twice a week, so the indexer runs on every change, not on a schedule, and the Function queues the re-index the moment a blob changes. The embedding runs on two replicas, because at 3,000 questions a minute one replica takes 140 percent of its capacity. The cache has a freshness rule: a cached billing answer expires when its article changes, and the Function clears it.
Trade-offs to name. A 90 percent cache rate means most customers get a stored answer, and a stored answer can go stale, so the freshness rule is load-bearing. The billing hard rule means one in five questions sometimes gets "let me get a person", and that costs a human. The higher TPM quota costs more at the peak, and the budget is met by the cache, not by the quota. Every one of those costs is visible, and every one is smaller than the cost of a chatbot that invents a refund policy.
One page with a table: one row per hop, naming the Azure service, what it does, what it costs, and what alert watches it. Plus the quota math for the peak, worked out in the open, and one sentence per failure saying what catches it. If a row cannot name its alert, the design is not done.
Saved in this browser, so the Lessons index can show your progress across the course.