Python is the language I reach for when I want a small job done fast. It reads almost like plain English, and it has a library for nearly everything, so I can write a script in an afternoon that would take a week in another language. AI work in Python means using that script to talk to a language model. A language model is a program that reads text and writes text back. It does not know your systems. It only knows the words you send it. So the job of the Python code is everything around the model: gather the logs, cut them down to size, count the cost, send the question, check the answer, and save what happened. Here is one concrete example. A pod restarts at 2 AM. I type one question into a small script: why did this pod restart? The script pulls the last hour of logs, picks the useful lines, asks the model, and prints back an answer with the log lines it used. That is Python for AI. The model is one hop. The Python around it is the other eight.
One question, one hour of logs, a hosted model over the network. The prompt fits the budget, the answer streams back, and the whole trip takes about 1,680 ms. This is the baseline every other scenario is measured against.
Every hop charges rent in milliseconds. The bar below is a worked example for one question, not a measurement from a live system, but the shape is honest: the model call and the streaming are most of the trip. The Python around them is fast and cheap. Step through the lesson or switch the scenario and watch the bar change. The local model triples the trip without changing a line of Python. The bloated prompt more than doubles it, and the answer gets worse at the same time.
The budget above is one fixed trip. This box is yours. Change the batch size and watch the token bill move. The workload is fixed: 200 log chunks for one question, 300 tokens of fixed overhead on every call, $0.005 per 1,000 tokens on the hosted model. The numbers are a teaching example, not a real vendor price. The shape is the lesson: the batch is the lever. Moving a control also updates the budget bar above, so you can see one call at your settings. The model call hop and the streaming hop scale with the tokens in the call.
How many log chunks go into one model call. A bigger batch means fewer calls, and the fixed 300 token overhead is paid fewer times.
How big each chunk is. Bigger chunks carry more context per call, but they eat the token budget faster.
The most tokens one call is allowed to carry. If a call needs more than this, the script refuses it instead of sending a prompt that will be cut off.
Hosted means the call goes over the network and costs per token. Local means the model runs on your own machine through Ollama: no token bill, but each call takes longer.
Try this: (1) Set the batch size to 1 at the defaults. The cost nearly doubles to $0.60 and the time jumps to 240 seconds, for the same 200 chunks. The only thing that changed is how many times the fixed overhead was paid. (2) Now set the batch size to 50 and raise the budget to 16,000. Four calls, 61,200 tokens, $0.31, 4.8 seconds. Same chunks again. The batch is the lever. (3) Switch the model to Local Ollama at the defaults. The cost drops to $0.00 and the time rises to 90 seconds. That is the trade, in one toggle.
Three questions. No score is saved anywhere — this is just you against the material.
Same topic, four times. First the Python around the model, then counting tokens and batching, then retrieval and evals, then the level where prompts are code and the run log is the audit trail.
Definition: AI work in Python means using a script to talk to a language model. A language model is a program that reads text and writes text back. It does not know your systems. It only knows the words you send it. So the job of the Python code is everything around the model: gather the logs, cut them down to size, count the cost, send the question, check the answer, and save what happened.
Types:
Uses: A pod restarts at 2 AM. I type one question into a small script: why did this pod restart? The script pulls the last hour of logs, picks the useful lines, asks the model, and prints back an answer with the log lines it used. That is Python for AI. The model is one hop. The Python around it is the other eight. I choose local when the data cannot leave the building, when the network cannot be trusted, or when I am testing and I want to run a thousand calls without watching a bill. I choose hosted when the answer has to come back in under a second for a user who is waiting.
Dimensions: One question and one hour of logs over a hosted model takes about 1,680 ms. The same question on a local model costs no tokens and takes far longer. As questions grow, the hosted bill grows per token and the local machine grows in power and waiting time. What breaks is a prompt that does not fit the budget, so the call is refused, or an answer with no sources, so I cannot tell whether the model read the logs or invented the answer. I notice the first from the refusal and the second by asking for the log lines in the answer. What it costs is per-token hosted spend against a machine, power, and my time for local. The trade is speed against privacy. Most real systems use both: local for the drafts and the tests, hosted for the answers users see.
Definition: Every model call carries a fixed cost: the system prompt, the question, the formatting. That cost is paid once per call, no matter how small the batch is. So a batch of one pays the fixed cost 200 times for 200 chunks, and a batch of fifty pays it four times. Token budgeting is counting before I send: a rough count of about one token per four characters, good enough for a guard, with the exact count from the model's own counter right before the real call.
Types:
Uses: I batch until the call is near the budget, then stop. Past the budget the call is refused, and before the budget the overhead is eating me. I keep the token estimate in the script and the exact count from the model's counter before the real call, because a refused call is a bug I can fix in daylight and a call that got cut off silently is a wrong answer at 2 AM. I validate the answer shape with pydantic the moment it arrives, so a model that answers in prose when I asked for JSON fails at the door, not in the report.
Dimensions: At ten times the questions, the first thing that breaks is the rate limit. I add a queue in front of the model calls, the same queue idea from Lesson 15, and I cache answers to questions I have already seen, because half of on-call questions are the same question asked twice. What breaks is a batch so big that the useful lines drown in noise, so the answers get worse even as the cost per chunk falls, or a budget guard that uses the rough estimate and trusts it past the real limit. I notice the first from eval scores that fall as the batch grows, and the second from a refused call I thought was under budget. What it costs is token spend, cache storage, and the time to measure the sweet spot instead of guessing it. The trade is batch size against answer quality. Bigger batches mean the model sees more context in one call, until the batch gets so big that the useful lines drown.
Definition: Retrieval means finding the useful lines before asking the model, instead of stuffing everything into the prompt. Embeddings turn text into vectors, numbers that capture meaning, and a vector store finds the lines closest to the question. An eval harness is a small test suite for my prompts: a file of questions I already know the answers to, plus the answers I expect. Every time I change a prompt, a chunk size, or a model, I run the harness and count how many answers still match.
Types:
Uses: I retrieve the right runbook page before asking, so the model answers from my systems instead of from its training data. I keep the eval cases in version control next to the prompts, because a prompt change without an eval run is a deploy without a test. If the score drops, the change was a step back, and I know it in a minute instead of finding out from a user at 2 AM. I keep the eval comparison strict about facts and loose about wording, because two wordings can carry the same fact, and only the fact matters on call.
Dimensions: At a hundred times the questions, the easy ones move to a small local model and the hard ones stay on the hosted model, and the cost becomes the design problem. What breaks is retrieval that returns the wrong page, so the answer is confident and wrong, or eval cases that rot and stop catching regressions. I notice the first when the answer cites a page that has nothing to do with the question, and the second when a bad prompt change ships with green evals. What it costs is embedding compute, vector-store upkeep, and the time to keep eval cases honest. The trade is retrieval quality against prompt size. Better retrieval means a smaller prompt, and a smaller prompt means a cheaper call with less noise.
Definition: A prompt is not a string in the middle of a script. It is a contract with the model, and it changes as often as code does. I keep prompts in their own files, versioned, with a name and a version number, and the script loads the prompt by name. Streaming is not a display trick. It is a debugging tool and a safety valve: when tokens arrive one by one, I can stop a bad answer early, and I can see exactly where it went wrong. And every model call needs a timeout, set from the p99 of the last hundred calls, not from a guess.
Types:
Uses: A prompt that tries to do five jobs does none of them well. One prompt, one job, and if the job needs two steps, I write two prompts and chain them, with a check between them. Prompts that live inline in code get edited in a hurry at 2 AM, and nobody remembers what changed, so mine live in files with versions. At 2 AM, streaming shows me where the answer went wrong while it is still arriving, and the run log answers the question someone will ask the next morning: what did the system say, and why.
Dimensions: A runaway script can burn a month of budget in a night, so I put a budget alert on the daily token spend. What breaks is a hosted call that hangs on a bad network, a local call that hangs while the model loads, or a prompt edited inline at 2 AM with no version and no eval run. I notice the first two from a timeout that fires and a retry that fails loudly, and the third only when the answers change and nobody can say what changed. What it costs is eval runs on every change, log storage, and the discipline of keeping prompts short and versioned. The trade is control against velocity. A refused call is a bug fixed in daylight. A silent cutoff is a wrong answer at 2 AM.
The nine hops above are the whole trip. This section is what I reach for once the trip works and I have to keep it working, day after day, without the bill or the answers drifting.
Batching is the cheapest optimization in this whole lesson, and it is also the easiest to get wrong. Every model call carries a fixed cost: the system prompt, the question, the formatting. That cost is paid once per call, no matter how small the batch is. So a batch of one pays the fixed cost 200 times for 200 chunks, and a batch of fifty pays it four times. The simulator above shows the math. The rule I follow is simple: batch until the call is near the budget, then stop. Past the budget the call is refused, and before the budget the overhead is eating you. And batching has a second effect nobody mentions. A bigger batch means the model sees more context in one call, so its answers get better, until the batch gets so big that the useful lines drown in noise. The sweet spot is where the tokens per call sit just under the budget and the chunks in one call are about one incident.
An eval harness is a small test suite for your prompts. It is a file of questions you already know the answers to, plus the answers you expect. Every time I change a prompt, a chunk size, or a model, I run the harness and count how many answers still match. If the score drops, the change was a step back, and I know it in a minute instead of finding out from a user at 2 AM. The harness is plain Python: a list of cases, a loop that calls the same function the real script calls, and a comparison that is strict about facts and loose about wording. I keep the eval cases in version control next to the prompts, because a prompt change without an eval run is a deploy without a test.
Streaming is not a display trick. It is a debugging tool and a safety valve. When tokens arrive one by one, I can stop a bad answer early, and I can see exactly where it went wrong. Without streaming, a bad answer costs the full call time before I see a single word. And every model call needs a timeout. A hosted call can hang on a bad network. A local call can hang while the model loads. A call without a timeout is a script that can hang forever, and a script that hangs forever is a page that never gets answered. I set the timeout from the p99 of the last hundred calls, not from a guess, and I retry once on a timeout, then I fail loudly.
A prompt is not a string in the middle of a script. It is a contract with the model, and it changes as often as code does. I keep prompts in their own files, versioned, with a name and a version number, and the script loads the prompt by name. That way a prompt change is a diff I can review, a rollback I can do, and an eval I can run. Prompts that live inline in code get edited in a hurry at 2 AM, and nobody remembers what changed. I also keep the prompt short. A prompt that tries to do five jobs does none of them well. One prompt, one job, and if the job needs two steps, I write two prompts and chain them, with a check between them.
I run local models on Ollama with Qwen for my k8s-ai-sre work, because the cluster data never leaves the machine. That is the real reason to run local: privacy, not price. The price argument is a trap. A local model is free per token, but it costs a machine, it costs power, and it costs my time when it runs slow. The hosted model is fast and costs per token. I choose local when the data cannot leave the building, when the network cannot be trusted, or when I am testing and I want to run a thousand calls without watching a bill. I choose hosted when the answer has to come back in under a second for a user who is waiting. Most real systems use both: local for the drafts and the tests, hosted for the answers users see.
At 10x the questions, the first thing that breaks is the rate limit. I add a queue in front of the model calls, the same queue idea from Lesson 15, and I cache answers to questions I have already seen. Half of on-call questions are the same question asked twice. At 100x, the cost is the design problem. I move the easy questions to a small local model, keep the hard ones on the hosted model, and I put a budget alert on the daily token spend, because a runaway script can burn a month of budget in a night. The eval harness becomes a gate in the pipeline from Lesson 33: no prompt ships until the evals pass. And the run log becomes the audit trail, because at 100x, someone will ask what the system said, and when, and why.
# ai_helper.py - batch chunks, guard the budget, call the model
MAX_TOKENS_PER_CALL = 4000
FIXED_OVERHEAD = 300 # system prompt + the question itself
PRICE_PER_1000 = 0.005 # teaching example rate, not a real price
def estimate_tokens(chunks):
# A rough count: about 1 token per 4 characters of text.
# Good enough for a budget guard. The exact count comes
# from the model's own counter before the real call.
return sum(len(c) // 4 for c in chunks) + FIXED_OVERHEAD
def answer_question(question, chunks, model_call):
batches = [chunks[i:i + 10] for i in range(0, len(chunks), 10)]
answers = []
for batch in batches:
tokens = estimate_tokens(batch)
if tokens > MAX_TOKENS_PER_CALL:
raise ValueError(
f"batch needs {tokens} tokens, budget is "
f"{MAX_TOKENS_PER_CALL}. Shrink the batch."
)
answers.append(model_call(question, batch))
return answers
# eval_harness.py - the same function, tested on known questions
EVAL_CASES = [
("why did pod api-7 restart?", "OOMKilled"),
("how many restarts in the last hour?", "3"),
]
def run_evals(answer_fn):
passed = 0
for question, expected in EVAL_CASES:
answer = answer_fn(question)
if expected in answer:
passed += 1
else:
print(f"FAIL: {question!r} -> {answer!r}")
print(f"{passed}/{len(EVAL_CASES)} evals passed")
return passed == len(EVAL_CASES)
What the snippet means: two files, one idea. The first file batches the chunks ten at a time, counts the tokens before every call, and refuses to send a batch that is over budget. Refusing is the point. A refused call is a bug I can fix in daylight. A call that got cut off silently is a wrong answer at 2 AM. The token estimate is rough, one token per four characters, because the exact count comes from the model's own counter right before the real call. The second file is the eval harness. It asks two questions I already know the answers to and checks that the expected fact shows up in the answer. It does not check wording. It checks facts. I run it after every prompt change, and if it fails, the change does not ship.
This one stayed with me because the system worked exactly as designed, and the answer was still wrong.
A small script answered on-call questions from a set of runbooks. It read the runbooks, cut them into chunks, turned each chunk into a small set of numbers called an embedding, and stored those numbers in a search index. An embedding is just a way to turn text into numbers so that similar meanings end up close together. When someone asked a question, the script searched the index for the closest chunks and sent those to the model. It worked well for months. Then one runbook was updated. The old rollback step was removed, because it had caused an outage, and a new step took its place. Nobody rebuilt the index. The embeddings still described the old runbook. During an incident that night, someone asked the script how to roll back, and it quoted the old step, the one that had been removed for causing an outage. The person followed it. The incident got worse before it got better.
Three things. First, the index rebuilds automatically whenever a runbook changes. A runbook edit now triggers a rebuild the same way a code change triggers a test run. No rebuild, no merge. Second, every answer now shows the date of the runbook chunk it came from, next to the answer, so a stale source is visible before anyone acts on it. Third, the eval harness got new cases on purpose: questions about runbooks that changed recently, so a stale index fails the evals instead of failing a person at 2 AM. The lesson I kept: a search index is a cache, and a cache without an expiry is a lie waiting for its moment.
This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real AI helper. Take a minute with it before you read my approach.
Design a Python helper that answers on-call questions for a team of 20 engineers. It reads the team's runbooks and the last hour of logs. It must answer in under 30 seconds and cost under $50 a month. The team handles about 500 incidents a year. Sketch the pieces, the token budget, and the guardrails.
I would ask five things before I sized anything. How many questions does the team actually ask in a month, and how many of them are repeats of the same question? How big are the runbooks, and how often do they change? How fresh must the log data be: last hour, last day, or since the incident started? What happens when the answer is wrong: who catches it, and what does a wrong answer cost here? And what data can never leave the building, because that decides whether the model runs local or hosted?
Start with the questions. 500 incidents a year is about 40 a month, and if each incident brings 5 questions, that is 200 questions a month. Each question needs about 10 chunks at 300 tokens, plus 300 tokens of fixed overhead, so about 3,300 tokens per question. 200 questions is 660,000 tokens a month. At the teaching rate of $0.005 per 1,000 tokens, that is $3.30 a month, far under the $50 budget, which leaves room for retries and for the eval harness running daily. Time is the tighter limit. A hosted call answers in about 1.5 seconds, so a question with one call is done in seconds. A local model takes about 5 seconds a call, still inside 30 seconds for one question, but not for ten questions in a row. So I cache the answers to repeated questions, because half of on-call questions are repeats, and the cache turns a 5 second call into a 40 ms lookup. I keep the run log for every question, because the $50 budget needs proof, not trust.
Hosted buys speed and costs per token, so I use it for the questions users are waiting on. Local buys privacy and costs time, so I use it for the eval harness and the drafts. Caching buys speed and money and costs freshness, so I expire cached answers when the runbook changes, because a stale cached answer is the incident from this lesson. A bigger chunk buys context and costs tokens, so I keep chunks small and let the search pick the right ones. And the guardrail is the check hop: an answer without a source never reaches the user, even if that means the helper sometimes says it does not know. Saying 'I do not know' is a feature. It is the only honest answer when the evidence is not there.
Saved in this browser, so the Lessons index can show your progress across the course.