Platform Engineering · Lesson 63

Agentic AI

An agent is a program that takes a goal in plain words and figures out the steps itself. You say 'clear the stuck deploy.' A script would run a fixed list of commands and stop, even if the first one failed. An agent reads the goal, picks a tool, runs it, looks at the result, and decides what to do next. Then it does it again. That loop is the whole idea. It exists because some jobs cannot be written down in advance. A stuck deploy might need a log check, or a pod restart, or a rollback, and you do not know which one until you look. I run a small agent like this at home on my own machine, with a local model on Ollama, to help me triage Kubernetes problems. It reads the pod status, runs kubectl, and tells me what it found. When it works, it feels like a junior on-call engineer who never sleeps. When it does not work, it loops. It retries the same failing command, burns tokens, and pages nobody. This lesson follows one loop, step by step, and shows you where the time goes, where the money goes, and where the guardrails have to sit.

Difficulty: Advanced · Know this first: Lesson 62, Lesson 57

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every loop charges rent in milliseconds and in tokens. The bar below is a worked example for one task, not a measurement from a live system, but the shape is honest: one loop is cheap, six loops of the same failing tool call cost six times as much and finish nothing. The tool step is the biggest slice of every loop, because the tool talks to the real system. The model steps are smaller, but they repeat, and repetition is what makes the bill. Step through the lesson or switch the scenario and watch the bar change. The runaway scenario multiplies the loop steps by six without the agent doing anything different. The guardrail scenario costs one extra loop and saves a production restart.

Extra picture · starts simple, builds to the full picture

The agent, as boxes

Starts from scratch with a goal, a model and one tool; adds the memory stores, then the budget caps; ends with the planner / worker / critic multi-agent split (Master: the agent as a service). I read the picture first. The words under each step are only labels.

Moving: a goalStops: 10From: GoalTo: Answer
Goalstep 1 of 10step 1 of 10
Model / plannerstep 2 of 10step 2 of 10
Tool registryallowed toolsallowed tools
Tool runnerstep 4 of 10step 4 of 10
Real systemkubectl, logs, APIkubectl, logs, API
Result as textstep 6 of 10step 6 of 10
Short-term memoryconversationconversation
Long-term memory storestep 8 of 10step 8 of 10
Budget capssteps, tokens, dollarssteps, tokens, dollars
Answerstep 10 of 10step 10 of 10
Architecture

    where the moving unit is nowalready visitednot reached yetRequest · the way thereResponse · the way back

    Extra picture · starts simple, builds to the full picture

    The loop that would not stop

    Starts from scratch with one clean loop; repeats the failing loop; ends with the cap stopping it and the trace explaining it (Master: measuring the agent - the cap is a failure, not a price). I read the picture first. The words under each step are only labels.

    Moving: a goalStops: 9From: GoalTo: Trace shows the loop
    Goalstep 1 of 9step 1 of 9
    Tool failsstep 2 of 9step 2 of 9
    Same tool tried againstep 3 of 9step 3 of 9
    Same errorstep 4 of 9step 4 of 9
    Step count climbsstep 5 of 9step 5 of 9
    Token bill climbsstep 6 of 9step 6 of 9
    Step cap hitstep 7 of 9step 7 of 9
    Run stops unfinishedstep 8 of 9step 8 of 9
    Trace shows the loopstep 9 of 9step 9 of 9
    Process

      where the moving unit is nowalready visitednot reached yeta step where something stops or breaksRequest · the way thereResponse · the way backFailure · the packet stops here

      Change the loop and watch the bill

      The budget above is three fixed stories. This box is yours. Move the sliders and watch the tokens, the cost, and the time change. Then ask the question I ask before I give any agent a tool: how much does one task cost, and what happens if it loops?

      5

      The highest number of loops the agent is allowed to run before the cap stops it. If the task needs more loops than the cap allows, the run ends unfinished.

      2

      How many loops the agent runs before it finishes or is stopped. This is the number that drives the bill.

      2,000

      How many tokens one loop costs, counting the plan, the tool pick, the check, and the stop decision. This is the per-call cost from Lesson 57, multiplied by the loops.

      $5

      The price of the model, in dollars per million tokens. A bigger model costs more per token. A local model on Ollama costs nothing per token, which is why I run my own agent at home.

      Try this: (1) Set the loops to 5 with the cap at 5. The run hits the cap and ends unfinished, even though the bill is small. The cap is a failure, not a price. (2) Now set loops to 20, tokens per step to 8,000, and price to $30. One task costs $4.80 and takes 19,790 ms. At 1,000 tasks a day that is $4,800 a day. Same agent, same tools. Only the loops changed.

      Check yourself

      Three questions. No score is saved anywhere. This is just you against the material.

      Basic to master

      I read this topic four times here. First as a loop that does a job, then as a loop with a memory and a leash, then as a loop I can measure, then as a service with a queue and a kill switch.

      Basic — A loop that does a job

      Definition: An agent is a loop. The model calls a tool, reads the result, decides the next step, and keeps going until the job is done or it gets stuck. The loop is the whole idea. The model is the brain. The tools are the hands. The loop is what turns one answer into a job done.

      Types:

      • A ReAct loop, thinking and acting in turns, flexible and slow, because every turn is a model call.
      • A plan-and-execute loop, planning the whole job up front, then running the steps, faster and more brittle.
      • A single agent, one loop, one job.
      • A multi-agent setup, a planner, a worker, and a critic that checks the work.

      Uses: I use the basic level for an on-call helper. A pod keeps restarting. The agent reads the pod status, reads the logs, reads the restart count, and writes a summary with the most likely cause. I run it on my own machine first, for my k8s-ai-sre project, where the model stays local and nothing leaves the machine.

      Dimensions: At one job at a time, the loop is cheap and the failure is small. What breaks is the loop that never stops: the same tool, the same error, twenty times, because nothing told it to quit. I notice it on the step count and on the token bill for one job. The cost is a few model calls per job. What I give up is the predictability of a script. A script does the same thing every time. An agent does what it decides, and sometimes it decides wrong.

      Intermediate — Memory and limits

      Definition: At this level I give the loop a memory and a leash. Short-term memory is the conversation in this run. It grows with every loop, and every token in it is paid again on the next call, so I trim it. Long-term memory is a store outside the run: past incidents, runbooks, notes the agent wrote last week. And the leash is the cap that stops a bad loop.

      Types:

      • Short-term memory, trimmed to the last few tool results in full, older ones summarized into one short note.
      • Long-term memory, dated, with expiry, because a runbook from last year that says restart the pod is wrong if the pod was replaced by a deployment.
      • A step cap, the maximum loops per job.
      • A same-error stop rule, the same tool plus the same error twice means stop, do not retry.

      Uses: I use this level when the agent starts looping. I keep the last few tool results in full and summarize the older ones. I put a date on every memory and let old ones expire. Memory without expiry is a trap.

      Dimensions: As the job gets longer, memory is the bill and the risk. A run that carries its whole history pays for the whole history on every call. What breaks is stale memory, an old note that sends the agent down the wrong path, and the loop that burns the budget before it burns out. I notice it on tokens per job and on the step count. The cost is the trimming work and the memory store. What I give up is the agent's full context. A trimmed memory is a smaller window, and a smaller window sometimes misses the clue.

      Advanced — Measuring the agent

      Definition: At this level I stop guessing whether the agent is good and start measuring it. I keep a set of golden tasks, real jobs from real on-call shifts, with a known right answer. Every change to the agent runs against that set, and I measure three numbers.

      Types:

      • Success rate, how many tasks finished correctly.
      • Cost per success, the token bill divided by the tasks that actually finished, not the tasks that started.
      • Steps per success, how many loops the finished tasks needed.
      • Guardrails outside the model, an allowlist of tools, an approval step before any write, a check on the tool inputs and on the model output.

      Uses: I use this level when a change to the agent needs a yes or a no. A change that raises success rate and doubles cost per success is not a win. It is a trade, and I say so out loud. The rule I never break is that the model does not grade its own homework. The guardrail is code I wrote, and the model cannot talk its way past it.

      Dimensions: As the task set grows, the evals are the only thing that tells me the truth, because models change under me. A model update can move the success rate ten points overnight. What breaks is the eval set that never changes, or the guardrail the model learned to route around. I notice it when the evals are green and the on-call shifts say otherwise. The cost is the golden set and the schedule that runs it. What I give up is the speed of shipping a change on a feeling. At this level a feeling is not a test.

      An agent that cannot finish will not stop on its own. It calls the same tool again, gets the same result, and calls it again. The bill grows and the user waits. I cap the steps and the cost per run, and I stop the run when either cap is hit.

      A step cap alone is not enough. Ten steps can all be the same tool with the same arguments. I watch for repeats. The same call three times in a row means the agent is stuck, and I stop it and hand the transcript to a human. The transcript is the debug log. Without it, I cannot tell what the agent was trying to do.

      Master — The agent as a service

      Definition: At this level the agent is no longer a helper. It is a service, and it needs everything a service needs: a queue so a burst does not start a thousand loops at once, rate limits per team, a cost budget per day with an alert at 80 percent, and a kill switch that stops all runs in one click. The loop stays the same. The controls around it grow.

      Types:

      • A queue in front of the agent, so a burst waits instead of starting a thousand loops.
      • Per-team rate limits and a per-day cost budget with an alert.
      • A kill switch per job type, one click, all runs stopped.
      • A model-change watch, because the evals run on a schedule and a model update shows up as a score change.

      Uses: I use this level when the agent serves a team and the token bill is the first thing that breaks at 10x. I cap the loops, I trim the memory, and I move the easy tasks to a smaller model or a script.

      Dimensions: At 100x the tasks, a stuck loop is not one bad run. It is a hundred bad runs at once, and the circuit breaker is the only thing that notices before the bill does. What breaks is the control that was never tested: a kill switch nobody clicked, a budget alert nobody set. I notice it on the per-day spend and on the queue depth. The cost is a service team and the discipline to test the kill switch before the night I need it. What I give up is the hobby. At this level the agent is infrastructure, and infrastructure gets a runbook, an owner, and a page.

      Some actions cannot be undone. Restarting a pod is fine. Deleting a namespace is not. I mark irreversible tools as gated, and a gated tool waits for a human approval before it runs. The approval has a timeout. If nobody answers in five minutes, the run stops, it does not proceed.

      The approval queue is a list a human can read, with the tool, the arguments, and the reason the agent wants to run it. An approval that says yes to a wall of JSON is not an approval. It is a click.

      Go deeper

      The lesson above is one loop through nine steps. Production is the rest of it. Here is what I actually spend my time on once the loop is drawn and the budget is written down.

      ReAct vs plan-and-execute

      A ReAct agent thinks and acts in turns. It reasons about the next step, calls a tool, reads the result, and reasons again. That is the loop in this lesson. It is flexible, and it is slow, because every turn is a model call. A plan-and-execute agent plans the whole job up front, then runs the steps. It is faster and cheaper, because the planning happens once. It is also brittle, because the first surprise breaks the plan. I use ReAct when the job is a debug, where the next step depends on what the last tool said. I use plan-and-execute when the job is a known runbook with a fixed shape. Most real agents are a mix. Plan the shape, ReAct inside the uncertain steps.

      Memory: short-term and long-term

      Short-term memory is the conversation in this run. It grows with every loop, and every token in it is paid again on the next call. I trim it. I keep the last few tool results in full and I summarise the older ones into one short note. Long-term memory is a store outside the run: past incidents, runbooks, notes the agent wrote last week. The agent reads it at the start and writes to it at the end. The risk is stale memory. A runbook from last year that says 'restart the pod' is wrong if the pod was replaced by a deployment. I put a date on every memory and I let old ones expire. Memory without expiry is a trap.

      Multi-agent systems

      One agent can do one job. A hard job gets split: a planner agent, a worker agent, and a critic agent that checks the work. The planner breaks the goal into steps. The worker runs the tools. The critic reads the result and says whether it is good enough. That is three model calls per turn instead of one, so the cost triples and the latency triples. The win is that each agent has a small job and a short prompt, so each one is more reliable. I use multi-agent when one agent keeps failing at one part of the job, not before. Three agents that agree with each other are no better than one agent that is right.

      Agent evals

      You cannot improve an agent you cannot measure. I keep a set of golden tasks: real jobs from real on-call shifts, with a known right answer. Every change to the agent runs against that set. I measure three numbers. Success rate: how many tasks finished correctly. Cost per success: the token bill divided by the tasks that actually finished, not the tasks that started. Steps per success: how many loops the finished tasks needed. A change that raises success rate and doubles cost per success is not a win. It is a trade, and I say so out loud. I also run the evals on a schedule, because models change under you. A model update can move the success rate ten points overnight, and the eval set is the only thing that tells you.

      Guardrails

      Guardrails are rules that sit outside the model. An allowlist of tools. An approval step before any write to production. A check on the tool inputs: is this the right namespace, is this the right cluster, is this a delete. A check on the model output: does the answer contain a secret, does it contain a command that was never approved. The rule I never break is that the model does not grade its own homework. The guardrail is code I wrote, and the model cannot talk its way past it. I test guardrails the way I test firewalls: by trying to get around them.

      What changes at 10x and 100x

      At 10x the tasks, the token bill is the first thing that breaks. I cap the loops, I trim the memory, and I move the easy tasks to a smaller model or a script. At 100x, the agent is no longer a helper. It is a service, and it needs everything a service needs: a queue so a burst does not start a thousand loops at once, rate limits per team, a cost budget per day with an alert at 80 percent, and a kill switch that stops all runs in one click. The loop stays the same. The controls around it grow.

      Agent loop config

      A loop budget is a config file, not a hope. This is the shape I use. The cap, the token budget, the tool list, and the rules are all written down before the agent runs:

      agent:
        name: oncall-helper
        max_steps: 5
        max_tokens_per_step: 2000
        stop_on_same_error: 2   # same tool + same error twice: stop, do not retry
      tools:
        allowlist:
          - get_pod_status
          - read_logs
          - get_deployment_status
        blocked:
          - restart_deployment   # needs human approval, never auto-run
          - delete_pod
      approvals:
        production_write: human_required
        read_only: auto
      budget:
        max_cost_per_task_usd: 0.50
        alert_at_usd_per_day: 100

      Five lines do the work. The cap bounds the loops. The same-error rule bounds the retries. The allowlist bounds the blast radius. The approval rule bounds the writes. The budget bounds the money. If any one of those lines is missing, I do not ship the agent.

      Catch the loop at the third repeat

      A short trace of tool calls, with a loop detector watching it. The detector counts steps, adds up the cost, and stops the run the moment the same call repeats three times in a row.

      Run it with python3 watch_the_loop.py. Then change the third call to describe_pod and run it again. The run must finish instead of stopping.

      Reading the run command

      python3 watch_the_loop.py has two parts and no flags. python3 is the program. It is the Python 3 interpreter, the thing that reads the file and runs it. watch_the_loop.py is the file to run. It is a plain argument, not an option. Python reads it top to bottom and runs it. Leave the filename off and Python waits for you to type code by hand. That is the interactive prompt, not a run.

      Three options I reach for all the time. --version prints the Python version and stops. I run it first on a new machine, so I know what I am running against before I trust the output. -u forces unbuffered output. Prints show up the moment they happen instead of sitting in a buffer. That matters when I pipe output into a log file and watch it live. Without it a crashed run can look like it printed nothing. -m runs a library module as a script. python3 -m json.tool runs the JSON tool that ships with Python, with no file of my own. Try them now.

      python3 --version
      python3 -u watch_the_loop.py
      python3 -m json.tool --help

      Run the script once plain and once with -u. The output is the same here. The script is short and finishes fast. The difference shows up on a long run. With -u each line appears as it is printed. Without it the lines arrive in a burst at the end. That is the check. If both runs print the same lines, the flag did its job.

      def watch_the_loop():
          trace = [("get_logs", "api-7"), ("get_logs", "api-7"), ("get_logs", "api-7"),
                   ("describe_pod", "api-7")]
          STEP_CAP, COST_CAP = 12, 0.50
          cost_per_call = 0.04
          repeat, steps, cost = 1, 0, 0.0
          verdict = "FINISH"
          for i, call in enumerate(trace):
              steps += 1
              cost += cost_per_call
              if i and call == trace[i - 1]:
                  repeat += 1
              else:
                  repeat = 1
              if repeat >= 3:
                  verdict = f"STOP: same call {repeat}x in a row at step {steps}"
                  break
              if steps >= STEP_CAP or cost >= COST_CAP:
                  verdict = "STOP: cap hit"
                  break
          print(f"steps used: {steps}, cost: ${cost:.2f}, verdict: {verdict}")
          assert verdict == "STOP: same call 3x in a row at step 3"
      
      
      watch_the_loop()
      

      Steps used: 3, cost: $0.12, verdict: STOP, same call 3 times in a row at step 3. The step cap is 12 and the cost cap is $0.50, and neither was hit. The repeat rule fired first, which is the point. The cheap rule catches the loop before the expensive caps do.

      One time it broke

      This one stayed with me because the agent did exactly what it was asked to do, and that was the problem.

      What broke

      An on-call helper agent was given a simple goal: clear the stuck rollout. The rollout was stuck because the new image would not start. The agent tried to roll it forward again. It failed. It tried again. It failed. It kept trying, every few minutes, all night. Each retry re-applied the same broken image, and each retry was a full loop: plan, tool pick, tool run, check, write, decide. By morning the model bill for one night was more than the team usually spent in a month, and the rollout was still stuck. The agent had also tripped the API rate limit on the cluster, so the morning shift could not run their own checks for twenty minutes. Nobody was paged, because the agent reported each retry as 'in progress.'

      Timeline

      • 22:40The rollout sticks. The agent is asked to clear it. It plans a retry of the same deploy.
      • 22:47First retry fails. Same error as before. The agent writes a note and loops.
      • 23:15Sixth retry fails. The step cap is set to twenty, so the loop keeps going. Nobody notices, because the agent only reports when it finishes.
      • 01:30The API rate limit trips. The agent's retries are now failing on rate limits too, and it treats those as new errors worth retrying.
      • 03:10The model bill crosses the daily budget. There is no alert on the budget, so nobody knows.
      • 07:05Morning shift finds the rollout still stuck, the API rate limited, and a trace with hundreds of identical retries.

      What changed after

      Four things. First, the step cap went from twenty to five, and hitting the cap now pages the on-call engineer instead of ending quietly. Second, a same-error rule: the same tool with the same error twice means stop and hand off, never retry. Third, a cost meter with an alert at 80 percent of the daily budget, and a hard stop at 100 percent. Fourth, the agent now reports progress every loop, not just at the end, so a run that is looping is visible in the first ten minutes, not the next morning. The agent still clears stuck rollouts. It just cannot do it a hundred times in the dark anymore.

      If you remember nothing else

      Design exercise

      This is the kind of question I would ask in an interview, and the kind I ask myself before I give any agent a tool. Take a minute with it before you read my approach.

      The prompt

      Design an on-call triage agent for a platform team. When a page fires, the agent reads the alert, checks the service, looks at the recent deploys and the logs, and writes a first summary for the engineer: what broke, what changed recently, and what to check first. It must answer in under two minutes. It must never restart, scale, or delete anything on its own. Sketch the loop, the tools, the guardrails, and the cost.

      Clarifying questions to ask first

      I would ask five things before I sized anything. What does the alert carry: a service name, a dashboard link, a runbook, or just a message. How fresh must the summary be, and can it read from the last incident on the same service. What counts as a write: is posting a summary to the incident channel a write, or only changes to the cluster. What happens when the agent cannot reach the logs or the deploy history: does it say so, or guess. And what is the failure we design for: the agent looping on a bad alert, the agent reading the wrong service, or the model bill spiking during a big incident.

      Rough sizing, worked

      Start with the budget from this lesson. One loop pass costs 985 ms and about 2,000 tokens. A triage job usually needs three loops: check the service, check the recent deploys, read the logs. That is 3 x 985 ms plus 90 ms for the answer, about 3,045 ms, and about 6,000 tokens. At $5 per million tokens that is $0.03 per page. The two-minute target is easy on time and the cost is small. Now size the bad day. A big incident fires 200 pages in an hour. If every page starts an agent run, that is 200 runs, 600 loops, 1.2 million tokens, $6.00 for the hour. That is fine. But if each run loops to the cap of five instead of finishing in three, the tokens double and the time per run goes past ten seconds. So I cap the loops at five, I cache the deploy history for five minutes so two hundred runs share one read, and I put the runs in a queue with a limit of ten at once. Little's Law for the sizing: 200 pages an hour is about one page every 18 seconds, and each run takes about 3 seconds, so one worker handles the load with room to spare. The queue is there for the burst, not the average.

      Trade-offs to name

      A read-only agent is safe and limited. It can tell the engineer what broke, but it cannot fix it, and the engineer still does the fix at 2 AM. Adding a restart tool buys speed and costs safety, so I would add it behind a human approval, never auto-run. A bigger model writes better summaries and costs more per token, so I would use the big model for the summary and a small model for the tool picks. Long memory buys better triage, because the agent remembers the last incident on this service, and costs tokens on every call, so I would keep the memory to the last three incidents and nothing older. And the summary is a draft, not a verdict. The engineer reads it, checks it, and owns the call. An agent that writes the summary and the engineer that trusts it blindly are the same failure.

      Saved in this browser, so the Lessons index can show your progress across the course.

      Next in this series
      Lesson 64 — Fine-Tuning, Distillation & Evaluation
      Coming next

      This lesson taught the agent loop. The next lesson teaches what happens before the loop starts: how a model is tuned for a job, how a big model is distilled into a small one, and how you evaluate whether any of it actually worked.