Platform Engineering · Lesson 201

Agents That Use Computers and Browsers

Some agents work by calling tools. These agents work by using the computer the way I do. A browser agent opens a real browser, reads the page, fills in the form, and clicks the button. A computer-use agent takes a screenshot of the desktop, finds the button in the picture, and moves the mouse to it. I use them for the jobs nobody built an API for, which is most of the jobs in a real company. This lesson follows one task through both kinds of agent: update the handover note, first on the team status page, then in the desktop notes app. It shows the loop every agent runs: observe, decide, act, verify. It shows where the loop breaks, which is almost always the click. And it shows the permission check that stands between an agent and the irreversible things, because an agent that can click can also pay, delete, and send, and it will, if you let it.

Difficulty: Advanced · Know this first: Lesson 195, Lesson 194

Run it as:

When it breaks, I check

One thing to remember

\1Every hop charges rent in milliseconds. The bar below is a worked example for one task, not a measurement from a live system, but the shape is honest: the browser and the typing are most of the trip, and the permission check is the cheapest hop that matters. In the normal trip, the nine hops add up to 1,745 ms. The browser open is 380 ms and the typing is 430 ms. The model calls, planning and grounding, cost 420 ms between them. Step through the lesson or switch the scenario and watch the bar change. The blocked plan costs 180 ms total, because the trip stops at the permission check and nothing runs. The grounding miss costs 2,345 ms, because one retry adds a second grounding and a second verify.\2

Run the task yourself

The budget above is one fixed trip. This box is yours. Change how many actions the task needs, change the screenshot size, change how strict the verification is, and turn human approval off, and watch the trade move: a faster task that misses more, or a slower task that catches its own mistakes. Then ask the question I ask before every agent deploy: what does a miss cost, and who catches it?

How many observe, decide, act rounds the task needs. One click is one action. Typing a sentence is one action. A task that files a form needs more actions than a task that reads a page.

How wide the screenshot is. A wide screenshot shows small buttons clearly, so the agent grounds better. It also takes longer to take and longer to read. A narrow screenshot is fast and blind.

How hard the verify step checks the result. At 0 the agent reads the screen back once and trusts it. At 100 it re-screenshots and re-reads until the result matches, and it refuses to report done until it does.

Approval on means a human says yes before any action that cannot be undone: pay, delete, send. It costs about 1,200 ms of waiting per task. Approval off is faster, and a wrong click can ship with no human in the way.

Total time per task5,096 msabout 5.1 seconds, most of it acting, not thinking
Grounding misses expected0.7 per taskclicks that land on the wrong spot, before verification
Bad actions that ship0.3 per taskmisses the verify step does not catch
Screenshots taken9 per taskone per action, one per retry, two for verification

Try this: (1) Push actions from 6 to 20 at the defaults. The task goes from about 5.1 seconds to about 11.9 seconds, and the expected misses climb from 0.7 to 2.2 per task. Long tasks do not just cost more time. They give the agent more chances to miss. (2) Now drop the screenshot width to 480. Each loop gets faster, but the misses climb to 1.2 per task, because the small buttons blur. (3) Now turn approval off. The task gets 1,200 ms faster, and nothing stands between a wrong click and the real world. That is the trade, and it is never free.

Check yourself

Three questions. No score is saved anywhere. This is just you against the material.

Basic to master

I read this topic four times here. First as an agent that uses the screen, then as a loop, then as grounding plus permissions, then as a fleet of agents.

Basic — An agent that uses the screen

Definition: A browser agent opens a real browser and uses the page the way I do. It reads the text, fills the form, clicks the button. A computer-use agent takes a screenshot of the desktop, finds the button in the picture, and moves the mouse to it. Both of them do jobs nobody built an API for, and both of them work by seeing the screen, not by calling an endpoint.

Types:

  • A browser agent, driven through Playwright, Selenium, or Puppeteer.
  • A computer-use agent, driven through screenshots and mouse and keyboard actions.
  • A task that crosses both, like this lesson: browser for the page, desktop for the app.

Uses: I use the basic level for one concrete job. Update the handover note. The browser agent reads the status page and fills the form. The computer-use agent opens the desktop notes app, clicks Edit, and types the summary. One task, two surfaces, and the agent crosses from one to the other in the plan.

Dimensions: At low load an agent like this is a script with a model in it, and it feels simple. What breaks is the button that moved, because the agent grounded from a picture taken before the move. I notice it when the verify step fails, or when nobody verifies and the task reports success without doing anything. The cost is a screenshot and a model call per action. What I give up is the speed and certainty of an API. The agent is slower than a script and less certain than a call, and that is the price of having no API.

Do this now: open any page in your own browser, find one button, and write down its name from the page. Then take a screenshot of your desktop, find one button in the picture, and write down where it is. The first is what a browser agent reads. The second is what a computer-use agent reads. That is the difference between the two kinds of agent, and you can feel it in your hands.

Try it: Basic

What is the difference between a browser agent and a computer-use agent?

Now do it: Do the drill at the end of this tier on your own machine: find one button name from a page, then find one button in a desktop screenshot and write down where it is.

Intermediate — The loop: observe, decide, act, verify

Definition: At this level the agent is not a list of clicks. It is a loop with four steps, and it runs the loop until the task is done or it has to stop. Observe: look at the screen or the page. Decide: pick the next action from what you saw. Act: do it. Verify: read the screen back and check the action did what it was supposed to do. Then loop.

Types:

  • Observe from a page tree, exact and fast.
  • Observe from a screenshot, the only option on the desktop.
  • Decide with a plan, or decide one action at a time.
  • Verify by reading the result back, or verify by checking the count.

Uses: I use this level when the first try misses. The agent clicks empty space, the verify step reads the screen back, the result does not match, and the loop runs again with a fresh screenshot. The retry is not a special case. It is the loop doing what the loop is for.

Dimensions: As tasks get longer, the loop is the cost and the safety net at once. Each round costs a screenshot and a model call, so a twenty action task costs twenty of each. What breaks is a loop with no limit: an agent that retries forever, clicking the same wrong spot, burning model calls. I notice it on the action count. The cost is time per round. What I give up is the single straight run. A loop is slower than a script that works first time, and it is the only thing that works when the screen changes under you.

Do this now: run the snippet from this lesson with python3 agent_loop.py and read the output. Every line is labeled OBSERVE, DECIDE, ACT, or VERIFY. Count how many times the loop runs. Then change the note text in the script and run it again. The loop is the same. Only the words change.

Try it: Intermediate

What are the four steps of the agent loop?

Now do it: Run python3 agent_loop.py from this lesson on your own machine and count how many times the loop runs. Every line is labeled OBSERVE, DECIDE, ACT, or VERIFY.

Advanced — Grounding and permissions

Definition: At this level the work is two hard problems, and they are not the model's problems. Grounding is turning a label into coordinates, and it fails when the layout moves, when two buttons look alike, or when the screenshot is too small. Permissions are deciding which actions the agent may run alone, and the answer lives in a list written in plain code, not in the model's judgment.

Types:

  • Coordinate grounding, where the model returns an x and a y.
  • Set-of-mark grounding, where numbered boxes are drawn on the screenshot and the model picks a number.
  • An allowlist, actions the agent runs without asking.
  • An approval gate, actions that stop and wait for a human.
  • A deny list, actions the agent never runs.

Uses: I use this level when the task can do real damage. Pay, delete, and send need a human yes, every time. The handover update does not, because it is reversible and the user asked for it. The line between the two is the permission list, and I write it before the agent runs, not after something goes wrong.

Dimensions: As the agent gets more tools, the permission list is the thing that keeps it safe, and the grounding is the thing that keeps it accurate. What breaks is a grounding model that is confident and wrong, because a wrong click looks like success. I notice it on the clicks that land on nothing, and on the verify failures. The cost is the dot check before irreversible clicks and the human wait on approvals. What I give up is speed. An agent that asks a human is slower than one that does not, and that slowness is the safety.

Do this now: open agent_loop.py, move click_save from ALLOWLIST to NEEDS_APPROVAL, and run it again. The agent stops before saving and prints that the action needs a human yes. One line moved, and the whole task stops. That is what a permission list does.

Try it: Advanced

Pay, delete, and send need a human yes every time. Where does that rule live?

Now do it: Open agent_loop.py on your own machine, move click_save from ALLOWLIST to NEEDS_APPROVAL, and run it again. Confirm the agent stops and asks for a human yes.

Master — Agents at fleet scale

Definition: At this level the agent is a service, not a script. A queue holds the tasks. A pool of browsers runs them, each with its own profile and its own login session. Every run writes a trace: what it saw, what it clicked, what the screen showed after. A human review inbox holds the actions that needed approval. The fleet is the unit now, not the run.

Types:

  • A task queue, so a hundred tasks do not start at once.
  • A browser pool, with cleanup after every run.
  • A trace per run, the flight recorder for the fleet.
  • An approval inbox, where a human says yes or no in one click.
  • A duplicate guard, so a retried task never files the same report twice.

Uses: I use this level when the task runs hundreds of times a day and the cost of a wrong click is measured in real money or real deletions. The trace is what I show the person asking why the agent did what it did. The approval inbox is what keeps the fleet from paying, deleting, or sending while nobody is watching.

Dimensions: At 100x, the screenshot bill is the first thing that hurts, and the second is the login session that outlives its task. What breaks is the browser that keeps yesterday's login, or the task that retries and files twice. I notice it on the duplicate count and on the screenshot storage. The cost is a queue, a pool, a trace store, and a human on call for approvals. What I give up is the simple script. At this scale the agents are infrastructure, and infrastructure gets an owner, a dashboard, and a page.

Do this now: take the simulator numbers at the defaults, 9 screenshots per task. Multiply by the tasks your team runs in a week. That number is your screenshot bill, and it is the first number to bring to the conversation about running agents at scale.

Try it: Master

A retried task in my agent fleet filed the same report twice. Which guard was missing?

Now do it: Take the simulator defaults, 9 screenshots per task, and multiply by the tasks your team runs in a week. Write that number next to the name of the person who approves the fleet's actions.

Extra picture · starts simple, builds to the full picture

The loop every agent runs

\1Starts from scratch with one look at the screen; adds grounding, then the permission gate, then the action; ends with the check that catches a wrong click (Master: the same loop at fleet scale). I read the picture first. The words under each step are only labels.\2
Moving: one taskStops: 5From: See the screenTo: Checked result
See the screenstep 1 of 5step 1 of 5
Ground the clicklabel to x, ylabel to x, y
Decide the actionallowed or notallowed or not
Do the actionclick or typeclick or type
Check the resultstep 5 of 5step 5 of 5
\1\2
Architecture

    where the moving unit is nowalready visitednot reached yeta step where something stops or breaksRequest · the way thereResponse · the way backFailure · the packet stops here

    Go deeper

    The lesson above is one task through nine hops. Production is the rest of it. Here is what I actually spend my time on once the loop works on my machine and I have to trust it on real work.

    A browser agent and a computer-use agent are two different tools

    A browser agent talks to the browser. It asks for the page tree, finds the button by name, and clicks it through the browser's own hooks. Playwright, Selenium, and Puppeteer all work this way. A computer-use agent talks to the screen. It takes a screenshot, finds the button in the picture, and sends a mouse click to the operating system. The browser agent is exact and fast on the web, because the page tells it where everything is. The computer-use agent is the only option on the desktop, because a desktop app has no page tree to ask. I pick by the surface. Web page, browser agent. Desktop app, computer-use agent. A task that crosses both, like this lesson, uses both, and the handoff between them is a step in the plan, not an accident.

    Grounding is the whole hard part

    Grounding is the step where the agent turns a label into coordinates, and it is where these agents fail most often. Three things make it fail. The layout moves, because apps ship updates and buttons change places. Two elements look alike, because Delete and Duplicate are the same size, the same color, and ten pixels apart. The screenshot is too small, because a 480 pixel wide picture turns a 12 pixel button into a blur. The fixes are boring and they work. Take a fresh screenshot right before grounding, never ground from a picture taken a minute ago. Ask the grounding model for the element and its neighbors, not just the element, so it can tell Delete from Duplicate by what sits next to it. And check the dot: draw the coordinates back onto the screenshot and confirm the dot is on the button before the click happens. A grounding model that returns coordinates and a confidence number is worth more than a bigger model, because a low confidence number is a signal to re-screenshot, and a confident wrong answer is the failure mode.

    One technique worth knowing by name is set-of-mark grounding. Instead of asking the model for raw coordinates, you draw numbered boxes on the screenshot first, one per element, and ask the model to pick a number. The model picks 7 instead of guessing x 1225, y 112. It is more reliable, because picking from a list is easier than estimating a position, and the numbers are cheap to draw. It costs one extra pass over the screenshot. I pay it on any screen where two buttons look alike.

    The permission model, written down

    The permission check in this lesson is a list, and the list lives in a file I can read. That is the whole design. The model never decides whether an action is safe. It proposes actions. Plain code checks them against the list. Here is the list from the runnable snippet in this lesson, in the same shape I use in production.

    allowlist:
      read_page: allowed        # read text from a page or a screen
      fill_form: allowed        # type into a field the user named
      click_save: allowed       # save a note the user asked to change
      click_edit: allowed       # open an edit box, change nothing yet
    needs_human_yes:
      submit_payment: approval  # money moves, always ask
      click_delete: approval    # deletion cannot be undone
      send_message: approval    # a message to a customer, once sent, is sent
    deny:
      run_shell_command: never  # an agent that clicks should never run a shell
      install_software: never   # nothing new on the machine without a human

    Read it top to bottom and it is the lesson in twelve lines. Allowed means the agent runs it without asking. Needs a human yes means the agent stops and waits, every time, even if the user said yes yesterday. Deny means the agent refuses and says so in plain words. I keep this file in version control, I review changes to it like code, and a new action defaults to needs a human yes until someone writes it into the allowed list on purpose. A permission list that grows by accident is a list that will eventually allow the wrong thing.

    When a browser agent beats an API, and when it does not

    A browser agent beats an API in exactly one case: there is no API. The team status page has no endpoint, the vendor portal has no webhook, the expense form lives behind a login page nobody owns. Then the browser is the tool, and it is a good tool, because it does what a person would do. Everywhere else, the API wins. It is faster, it is deterministic, it never misses a button, and it costs a fraction of the model calls. The test I use is simple. If the same data comes back from an API call in 50 ms, I do not send an agent with screenshots to fetch it in 2 seconds. The browser agent is the fallback, not the first choice. And the browser agent has walls the API does not: login pages, CAPTCHAs, and rate limits that treat an agent like a bot, because to the site, it is one. An agent that hits a CAPTCHA stops and asks a human. It never tries to solve it. That is a policy, not a technical limit.

    What changes at 10x and 100x

    At 10x the tasks, the first thing that breaks is the screenshot bill. Ten screenshots per task at 500 tasks a day is 5,000 screenshots a day, and each one is a model call to read. The fix is to screenshot less: read the page tree where it exists, screenshot only the desktop steps, and keep the screenshots small enough to read but no bigger. The second thing that breaks is the browser pool. One browser per task is fine at ten tasks a day. At a hundred, you need a pool of browsers, each with its own profile, its own login session, and its own cleanup, because a browser that keeps yesterday's login is a browser that can do yesterday's damage. At 100x, the agents become a service of their own, with a queue, a trace per run, and a human review inbox for the actions that needed approval. The trace matters most. Every run writes down what it saw, what it clicked, and what the screen showed after. When an agent deletes the wrong thing at 100x, the trace is the only way to know what it saw, and the trace is what you show the person asking why.

    The loop, in code

    The snippet below is the loop from this lesson, run for real in this VM. It uses a mock browser driver with the same verbs Playwright uses, goto, fill, click, and text, against a local page, plus a mock desktop screen that grounds labels to coordinates. It runs the handover update in both places, verifies both, and then refuses a pasted instruction to pay. The full file is agent_loop.py in the lesson work log.

    ALLOWLIST = {"read_page", "fill_form", "click_save", "click_edit"}
    NEEDS_APPROVAL = {"click_delete", "submit_payment", "send_email"}
    
    def permission(action):
        if action in NEEDS_APPROVAL:
            print("  PERMISSION: %s needs a human yes. STOPPING." % action)
            return False
        if action in ALLOWLIST:
            return True
        print("  PERMISSION: %s is not on the allowlist. STOPPING." % action)
        return False
    
    # observe -> decide -> act -> verify
    browser.goto(base_url)                  # OBSERVE the handover page
    browser.fill("#note", new_note)         # ACT, allowlisted
    browser.click("#save-btn")
    x, y = screen.ground("Edit note")       # GROUND the label to (1225, 112)
    screen.click(x, y)
    screen.type_text(new_note)
    assert browser.text("#saved") == new_note   # VERIFY, both places
    assert screen.note == new_note
    assert permission("submit_payment") is False  # pasted "click Pay now": refused

    Run it with python3 agent_loop.py. The output ends with AGENT RUN: PASS. The browser half ran for real too, with Playwright driving a real Chromium against the same local page: goto, fill, click, read it back, screenshot 20,763 bytes, PLAYWRIGHT RUN: PASS. The browser download is the only part that needs the network. The loop itself needs nothing but the local page.

    Reading the run command

    python3 agent_loop.py has two parts and no flags. python3 is the program. It is the Python 3 interpreter, the thing that reads the file and runs it. agent_loop.py is the file to run. It is a plain argument, not an option. Python reads it top to bottom and runs it. Leave the filename off and Python waits for you to type code by hand. That is the interactive prompt, not a run.

    One option I reach for all the time. -u forces unbuffered output. Prints show up the moment they happen instead of sitting in a buffer. That matters when I pipe output into a log file and watch it live. Without it a crashed run can look like it printed nothing.

    python3 --version
    python3 -u agent_loop.py

    Run the script once plain and once with -u. The output is the same here. The script is short and finishes fast. The difference shows up on a long run. With -u each line appears as it is printed. Without it the lines arrive in a burst at the end. That is the check. If both runs print the same lines, the flag did its job.

    One time it broke

    This one stayed with me because the agent reported success. Every dashboard was green. The run log said all four reports were filed. One of them was deleted instead.

    What broke

    An agent that filed weekly expense reports clicked Delete instead of Duplicate. The two buttons sat side by side in the portal, same size, same color, ten pixels apart. A portal update had swapped their order the night before. The agent grounded Duplicate from a screenshot taken before the update, clicked Delete, and the draft report vanished. It filed the next three reports, verified each one, and reported success. The deleted draft was never filed, and nobody noticed for a week, because the agent's report said all four reports were done.

    Timeline

    • 09:00Agent starts the weekly expense run, four drafted reports.
    • 09:02First report: agent clicks Delete instead of Duplicate. The draft is gone. The agent does not notice, because the click landed and the portal showed no error.
    • 09:04Agent files the remaining three reports, verifies each one, reports all four done.
    • 09:10The person who owns expenses opens the portal, sees three reports, and assumes the fourth is still a draft.
    • Next MondayFinance asks where the fourth report is. The draft is gone, the receipts have to be gathered again, and the week of expenses is filed late.
    • Same dayThe run log shows the click at (1060, 640). The Delete button sits at (1060, 640) in the new layout. Duplicate moved to (1180, 640).

    What changed after

    Four things. First, the agent takes a fresh screenshot right before grounding, never from a picture taken before the run started. Second, the dot check: the agent draws its coordinates back on the screenshot and a second, cheap check confirms the dot is on the labeled button before any irreversible click. Third, delete is now a needs-a-human-yes action, even inside a run the user already approved. Approval for a run is not approval for a deletion. Fourth, the verify step reads back the count, not just the last report. Four reports in, four reports filed, or the run reports a failure. A verify step that checks the thing it changed is a check. A verify step that checks the last thing it did is a habit.

    If you remember nothing else

    Design exercise

    This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real agent. Take a minute with it before you read my approach.

    The prompt

    Design an agent that files the weekly expense report for 500 employees through a web portal that has no API. Each report needs a receipt upload, a category pick, and a submit. A report must never be submitted twice, and a wrong category costs a real correction. Sketch the loop, the permission gates, and the budget.

    Clarifying questions to ask first

    I would ask five things before I sized anything. What counts as irreversible here: the submit, the category pick, or both? Who approves, the employee or a manager, and do they approve one report at a time or a batch? What happens on a duplicate: does the portal reject it, or does it file twice and charge twice? What do the receipts look like: photos, PDFs, one file or five? And when does the peak hit: all 500 on Friday afternoon, or spread across the week?

    Rough sizing, worked

    Start with the loop from this lesson. Each report is about 8 actions: open the portal, open the draft, upload the receipt, pick the category, check the total, submit, read it back, report. Each action is one loop of about 440 ms at the simulator defaults, so one report costs about 3.5 seconds of acting, plus a verify of about 170 ms, plus a human approval wait of about 1,200 ms if the submit needs a yes. Call it 5 seconds per report. Sequential, 500 reports take about 40 minutes. That is fine for a weekly job, until Friday afternoon arrives and all 500 land at once. So I run 10 browsers in parallel, each with its own profile and login, and the wall clock falls to about 4 minutes. Little's Law for the sizing: 500 reports at 5 seconds each is 2,500 seconds of work, and 10 workers clear it in 250 seconds. The duplicate guard is a key per employee per week, checked before the submit, not after. The portal has no idempotency, so the agent keeps its own record of what it filed, and a retried task reads that record first. The screenshots are the evidence: about 10 per report, 5,000 a week, kept for 30 days, then deleted.

    Trade-offs to name

    A browser agent buys you a working system today and costs you a miss rate on every run, because the portal will move a button one Friday and nobody will tell you. Waiting for an API buys certainty and costs months. Approval per report buys safety and costs a human afternoon, so I approve the submit in a batch: the agent prepares all 500, a human reviews the exceptions, and the clean ones submit. Parallel browsers buy speed and cost rate limits, because the portal will treat ten logins at once like an attack, so I stagger the starts and keep the pool small. And the verify step is the whole design. An agent that files 500 reports without reading them back will file 500 reports, and some of them will be wrong, and the first person to notice will be Finance.

    Saved in this browser, so the Lessons index can show your progress across the course.

    Next in this series · Lesson 202
    Fine-Tuning, Distillation & Evaluation

    A base model is a smart new hire who has read the whole internet and never worked at your company. Fine-tuning is training that hire on your own examples until it answers the way your team answers.

    Go to Lesson 202 →

    Or browse the full course from the Lessons index.