Platform Engineering · Lesson 164

Multimodal Models

A text-only model takes one thing in. Words. I paste a screenshot of a dashboard, I type a question, and I add a ten-second voice note, and the model has to use all three. The picture becomes tokens. The voice becomes tokens. The words become tokens. Then the three lists are fused into one, the model reads the picture and the runbook together, and the answer comes back with its sources. This lesson walks one screenshot through that pipeline, hop by hop, and shows the two ways it goes wrong: a 4K photo that costs sixteen times the tokens, and a blurry photo the model misreads. I learned this the way I learn most things. I trusted an answer about a picture the model had read wrong, and a team spent 40 minutes on the wrong service. This lesson is the check I wish I had run first.

Difficulty: Advanced · Know this first: Lesson 56, Lesson 60

Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for one screenshot and one question, not a measurement from a live system, but the shape is honest: on the normal path, generating the answer is more than half the trip, and the image encoder is the next biggest slice. The text is the cheapest part of a request that is mostly picture. Step through the lesson or switch the scenario and watch the bar change. The heavy path is twice as slow because the picture is sixteen times the tokens. The broken path costs almost the same as the normal one, because the failure is cheap when the check catches it.

Make the request heavier, and watch the bill move

The budget bar above shows three fixed stories. This simulator lets you write your own. It answers one question: what does this request cost in time, tokens, and money. You set the screenshot width, the voice note length, the video frames, and the retrieved chunks. Move a slider and watch the token counts, the per-hop timings, the budget bar, and the verdict change. The numbers are a teaching example, built from the formulas below, not live measurements. The shape is the lesson: image tokens grow with the square of the width, and every token costs time in the generator.

Image encoder 576 tokens Audio encoder 250 tokens Text tokenizer 12 tokens Fusion 838 tokens, one list Generator reads, then writes Grounded answer checked, with sources
1,024 px

Width of the screenshot in pixels. The height follows at 16 by 9. Image tokens are one per 32 by 32 patch, so doubling the width quadruples the tokens. 512 is a phone photo. 4,096 is a full 4K shot.

10 s

Seconds of voice note, sampled at 16,000 times a second. About 25 audio tokens per second. Zero means no voice note, and the audio hop is skipped. 120 seconds is a two-minute ramble.

0 frames

Frames taken from a video clip, each one encoded like a small 512-pixel picture at 256 tokens and about 150 ms. Zero is the normal request. 30 frames is one second of video at 30 frames a second.

5 chunks

Runbook chunks fetched at hop 7, 400 tokens each. Zero means the model answers from the picture alone, with no runbook. Ten is a long context and a slow read.

Try this: (1) Push the screenshot width to 4,096 at the defaults. Image tokens jump from 576 to 9,216, total time passes 6 seconds, and the verdict flips to slow. Same graph, same answer. Only the width changed. (2) Now set the voice note to 0 seconds. The audio hop is skipped, the total falls by 180 ms, and the answer loses the one sentence that said when the spike started. Ask yourself what that sentence was worth. (3) Set video frames to 30. That adds 7,680 tokens and about 4.5 seconds of encode time. A one-second clip costs more than the whole normal request. That is why video is sampled, never sent whole.

Formulas, in plain terms. Image tokens = (width / 32) x (height / 32), height = width x 9 / 16. Image encode ms = 20 + image tokens x 0.52. Audio tokens = seconds x 25. Audio encode ms = 0 if no audio, else 60 + seconds x 12. One video frame = 256 tokens and 150 ms. Total input tokens = image + audio + frames + 40 text + chunks x 400. Fuse ms = 30 + total tokens x 0.005. Retrieve ms = 60 + chunks x 30. Generate ms = 400 + total tokens x 0.19. Cost = input tokens x $3 per million + 300 output tokens x $15 per million. Moving a slider updates the hop timings, the budget bar, and the step panel above.

Check yourself

Three questions. No score is saved anywhere — this is just you against the material.

Basic to master

Multimodal is not one thing. It is four levels of the same job, and every team I have worked with sits on one of them. The levels are not about how smart the model is. They are about how many kinds of input the system takes seriously, and how much of the pipeline is built to check them. Read them in order. Each level fixes the failure that hurt the most at the level below it.

Basic — Text in, text out, pictures as attachments

At the basic level, the model reads text and the pictures are attachments a human opens. Somebody pastes a screenshot into the chat, and a person looks at it while the model answers the typed question. The model never sees the picture. That is the whole system. It works when the human is the vision model, and it breaks the moment the human is asleep.

Uses: A demo for one stakeholder. A support queue where a person reads every screenshot anyway. I use this level to learn what questions people ask about pictures, before I build anything that answers them.

Failure modes: The model answers a question about a picture it never saw, and the answer is a confident guess. The human stops checking, because the model sounds sure.

Intermediate — One vision model, one picture at a time

At the intermediate level, the screenshot goes to the model. The image encoder turns it into tokens, the model reads them, and the answer quotes the picture. One picture per request, downscaled to 1024 pixels, with a file check at the door. The grounding hop appears here: every number in the answer must exist in the picture or in a retrieved chunk.

Uses: A triage assistant that reads one dashboard screenshot and one question. An on-call helper that reads an alert image and points at the runbook.

Failure modes: A blurry photo gets a confident wrong answer, because the grounding hop is missing or the picture is never checked. A 4K photo makes 9,216 tokens and the bill jumps.

Advanced — Three inputs, one fused request

At the advanced level, the request carries a picture, a voice note, and text, and the pipeline fuses all three. Retrieval searches pictures as well as words, so an old screenshot with the same spike shape comes back next to the runbook page. Audio is sampled, encoded, and capped. Video is sampled by frames, never sent whole. The token budget is enforced at fusion, and the cost per request is on the dashboard.

Uses: The lesson you just walked. An incident assistant that takes a screenshot, a voice note from the person on call, and a question, and answers with sources.

Failure modes: The context window fills up and fusion cuts the picture silently. The audio encoder hears noise as words. The image search matches colors instead of shapes.

Master — Multimodal as a budgeted system

At the master level, every modality owns a budget in tokens, milliseconds, and dollars, and the budgets are in version control. Pictures are downscaled and cropped by rule. Audio is transcribed once and stored, not re-encoded on every request. Video is keyframes only. The eval set has pictures in it, including blurry ones, and the eval scores the grounding hop, not just the answer. At 10x the volume, the image encoder is a shared service with its own cache. At 100x, the screenshots are embedded once at ingest and the index answers the picture search without calling the encoder at all.

Uses: A platform that answers picture questions for a whole support org, with a cost dashboard and a grounding score on every answer.

Failure modes: The eval set has only clean pictures, so the blurry ones fail in production. The shared encoder becomes a single point of failure, and every picture question in the company goes down with it.

Go deeper

The image encoder in this lesson is a vision model trained on pictures paired with words. That pairing is the whole trick, and it has a name you will meet everywhere: contrastive training. The model sees a picture and a caption together, and it learns to put them close to each other in the same number space. A picture of a latency graph and the words latency graph end up near each other. A picture of a cat ends up far away. Once pictures and words live in the same space, the model can compare them. That is how the retrieval hop finds an old screenshot that looks like your screenshot. It is not matching pixels. It is matching meaning, in numbers.

Text and pictures do not tokenize the same way, and the difference decides your budget. Text is split by a tokenizer that learned which letter groups are common. Pictures are cut into squares. A 1024 by 576 picture makes 576 tokens at one token per 32 by 32 patch. A 4K picture makes 9,216. That is sixteen times the tokens for four times the width, because the tokens grow with area, not with width. Audio is sampled in time. 16,000 samples a second becomes about 25 tokens a second. Video is pictures in time, so a video is the most expensive input you can send. One second at 30 frames is 30 pictures. Nobody sends a whole video. Everybody samples keyframes, one frame every few seconds, and accepts that the motion between them is lost.

The failure modes are specific, and they are not the text failure modes from Lesson 56. A model can misread a digit on a blurry axis, and the answer will be wrong with full confidence. A model can read the wrong panel on a dashboard with six panels, because nothing told it which panel the question was about. A model can describe a picture that is not there, when the picture is dark or cropped, because it fills in what it expects to see. The fix for all three is the same shape. Crop the panel before you send it. Ask the question about one panel, not the whole dashboard. And ground every number, because a grounded wrong number is caught and a confident wrong number is shipped.

Multimodal RAG changes the index, not just the query. A text index holds chunks. A multimodal index holds chunks and picture embeddings side by side. When a screenshot arrives, the pipeline embeds it and searches both. The runbook page comes back for the words. The old screenshot comes back for the shape. The two together are stronger than either alone. The old screenshot from last month with the same spike shape told the model more than the runbook did, because the runbook says what to do and the picture says what it looks like when it happens. The cost of this index is paid once, at ingest. The screenshot is embedded when it is stored, not when it is asked about. A search that embeds the query picture on every request pays the encoder cost twice.

At 10x the traffic, the image encoder becomes the first problem. It is the slowest hop after generation, and every request pays it. I cache it. The same screenshot, the same resize, the same tokens. A dashboard screenshot that ten people paste in one morning is encoded once. At 100x, the encoder becomes a service of its own, with its own queue, its own autoscaling, and its own budget. The generator stops calling it inline. The request flow becomes: encode, store the tokens, then generate from the stored tokens. The bill also changes shape. At 10x, the cost per request is the number to watch. At 100x, the cost per thousand requests is the number, and the downscale rule is the lever that moves it.

Price it yourself: the token and cost math

Write the script below as multimodal_budget.py and run it with python3 multimodal_budget.py. It prices one request with the lesson's own stated assumptions: one image token per 32 by 32 patch, 25 audio tokens per second, one video frame at 256 tokens, 40 text tokens for the question, 400 tokens per retrieved chunk, 300 output tokens, $3 per million input tokens and $15 per million output tokens. No model is called. This is token and cost math, and it runs on this machine with plain Python.

import math

def budget(img_w=1024, audio_s=10, frames=0, chunks=5):
    h = round(img_w * 9 / 16)
    img_tokens = math.ceil(img_w / 32) * math.ceil(h / 32)
    img_ms = 20 + img_tokens * 0.52
    audio_tokens = audio_s * 25
    audio_ms = 0 if audio_s == 0 else 60 + 12 * audio_s
    frame_tokens, frame_ms = frames * 256, frames * 150
    total_tokens = img_tokens + audio_tokens + frame_tokens + 40 + chunks * 400
    fuse_ms = 30 + total_tokens * 0.005
    retrieve_ms = 60 + 30 * chunks
    generate_ms = 400 + 0.19 * total_tokens
    total_ms = 15 + img_ms + audio_ms + 8 + fuse_ms + retrieve_ms + generate_ms + 60
    cost = total_tokens * 3 / 1_000_000 + 300 * 15 / 1_000_000
    return img_tokens, audio_tokens, frame_tokens, total_tokens, total_ms, cost

for label, kw in [("default request", {}),
                  ("phone photo only", dict(img_w=512, audio_s=0)),
                  ("4K + 60s voice + 30 frames", dict(img_w=4096, audio_s=60, frames=30))]:
    it, at, ft, tt, ms, c = budget(**kw)
    print(label, "| image", it, "tokens | audio", at, "| frames", ft,
          "| total", tt, "tokens |", round(ms), "ms | $%.4f" % c)
print("500 requests a day, default shape: $%.2f a month" % (budget()[5] * 500 * 30))

Expected output, exactly as it ran in this build:

default request | image 576 tokens | audio 250 | frames 0 | total 2,866 tokens | 1781 ms | $0.0131
phone photo only | image 144 tokens | audio 0 | frames 0 | total 2,184 tokens | 1244 ms | $0.0111
4K + 60s voice + 30 frames | image 9,216 tokens | audio 1,500 | frames 7,680 | total 20,436 tokens | 10300 ms | $0.0658
500 requests a day, default shape: $196.47 a month

The script lands at 1,781 ms for the default request. The diagram rounds each hop to the nearest 5 ms and lands at 1,788 ms. Same shape, same story. The difference is rounding, not a different model. Check it yourself. Change the default width to 2048 and run it again. The image tokens go from 576 to 2,304, the total passes 3,500 tokens, and the cost per request nearly doubles. Then set the voice note to 0 and the chunks to 0. The total falls to 760 tokens, and the answer has no runbook and no voice note behind it. That is the trade in one number. Cheap and thin, or complete and slow.

Do it yourself: turn a picture into tokens

Run this in this VM with python3. It draws a fake dashboard screenshot with PIL, resizes it to 1024 by 576, and counts the patches. No model is called. The count is the lesson. A picture is not one input. It is 576 tokens.

python3 -c "
from PIL import Image, ImageDraw
import math
img = Image.new('RGB', (1920, 1080), (24, 26, 32))
d = ImageDraw.Draw(img)
d.rectangle([40, 40, 1880, 180], fill=(38, 42, 54))
d.text((70, 90), 'checkout p95 latency', fill=(240, 234, 217))
img.save('/tmp/n3/dashboard.png')
small = img.resize((1024, 576), Image.LANCZOS)
patches = math.ceil(1024/32) * math.ceil(576/32)
print('resized to', small.size, '-> patches:', patches, '(= image tokens)')
print('a 4K shot would make', math.ceil(4096/32) * math.ceil(2304/32), 'tokens')
"

Expected output, exactly as it ran in this build:

resized to (1024, 576) -> patches: 576 (= image tokens)
a 4K shot would make 9216 tokens

Now the audio side, same idea, no model. Ten seconds of voice at 16,000 samples a second is 160,000 samples and about 250 tokens. Run it and check the numbers yourself.

python3 -c "
for s in (0, 10, 60):
    print(s, 'seconds ->', 16000*s, 'samples,', s*25, 'audio tokens')
"

Expected output:

0 seconds -> 0 samples, 0 audio tokens
10 seconds -> 160000 samples, 250 audio tokens
60 seconds -> 960000 samples, 1500 audio tokens

One time it broke

This one stayed with me because the model did nothing wrong by its own rules. It read the picture, it wrote an answer, and the answer was wrong in a way nobody checked until the wrong team had spent 40 minutes on it.

What broke

It was a weeknight, about 1 AM. An alert fired for a checkout service, and the person on call pasted a screenshot of the dashboard into the team channel, next to a typed question: which service is actually slow. The assistant read the screenshot and answered with confidence. It said the payments service was slow, and it quoted a number to prove it. The payments team got paged. They spent 40 minutes looking at a service that was fine. The slow service was the checkout service itself. The assistant had read the number from the wrong panel. The dashboard had six panels, and the number it quoted belonged to a panel two rows down. The picture was small, the panels looked alike, and nothing in the request said which panel the question was about.

Timeline

  • 01:02Alert fires. The person on call pastes a full-dashboard screenshot and asks which service is slow.
  • 01:04The assistant answers: the payments service, p95 at 2,400 ms. The number is real. It comes from the wrong panel.
  • 01:10The payments team is paged. They open their own dashboard. Everything looks normal.
  • 01:35Somebody opens the original screenshot, zooms in, and reads the panel titles. The 2,400 ms number belongs to a different panel. The checkout panel says 3,100 ms.
  • 01:42The checkout team is paged instead. They find a stuck queue in ten minutes. The incident ends at 02:20.
  • Next dayThe team reads the incident back and finds the 40 minutes went to a panel title nobody read.

What changed after

Three things, and they are the same three things this lesson teaches. First, crop the panel before you send the picture. A question about one panel gets one panel, not six. A model that sees six similar panels will pick the wrong one, and it will sound sure about it. Second, the answer must name the panel, not just the number. A number without a panel title is a number without a source, and the grounding hop now rejects it. If the answer says 2,400 ms, it has to say which panel the 2,400 ms came from, and the panel title has to be in the picture. Third, a human reads the number before a team gets paged. The assistant points. The human confirms. That rule cost the team two minutes on every page, and it has saved them from paging the wrong team three times since. The model was not broken. The request was under-specified, the picture was too wide, and nobody checked the source. The fix was not a bigger model. It was a smaller picture and a stricter check.

If you remember nothing else

Design exercise

Design a screenshot triage assistant for an on-call team. It must read 500 dashboard screenshots a day, each with a typed question, and answer in under 5 seconds at p95. Each answer must name the panel it read and cite a runbook page. The monthly model bill must stay under $250. You have a vision model, a text model, a runbook, and a dashboard that stores old screenshots. Sketch the pipeline: the encoders, the token budget, the retrieval, and the grounding check.

Clarifying questions to ask

  • How big are the screenshots when they arrive: phone photos of monitors, or clean exports, and who downscales them before the encoder sees them?
  • Does the question point at one panel, or does the model have to find the panel itself, and what happens when it picks the wrong one?
  • What counts as a good answer: a number, a cause, or a next step, and who judges it when the picture is blurry?
  • How spiky is the traffic: is 500 a day the average or the peak, and what happens during an incident when ten people paste the same screenshot at once?
  • What is the failure we design for: the model misreading a digit, the bill tripling, or the answer arriving after the incident is over?

Rough sizing, worked

Start with the tokens. One screenshot downscaled to 1024 by 576 is 576 image tokens. The question is 40 text tokens. Five runbook chunks are 2,000 tokens. One request is about 2,616 tokens in and 300 out. At $3 per million input and $15 per million output, that is about $0.0123 a request. 500 requests a day is about $6.15 a day, about $185 a month. Inside the $250 budget, with room for a bad week. The time math is the other half. Check 15 ms, encode 320 ms, tokenize 8 ms, fuse 44 ms, retrieve 210 ms, generate 900 ms, ground 60 ms. That is about 1.6 seconds, well inside 5 seconds at p95, as long as nobody sends a 4K photo. A 4K photo makes 9,216 image tokens, and the same request becomes 10 seconds and $0.06. The downscale rule is what keeps the design inside both budgets. The cache is the third lever. Ten people pasting the same screenshot during an incident is one encode, not ten, because the screenshot is hashed at the door and the tokens are stored. The storage math is small. 500 screenshots a day at 20 KB each is 10 MB a day, 300 MB a month, and the token cache is the thing that saves the money, not the picture store.

Trade-offs to name

Downscaling buys cost and speed and costs legibility, so I downscale to 1024 and crop the panel first, and I reject a photo the encoder cannot read instead of guessing at it. Five chunks buy grounding and cost context, so I fetch five and I show the two the answer actually used. Caching the encode buys speed during an incident and costs a cache to run, so I cache by picture hash for 24 hours and say so in the runbook. A strict grounding check buys trust and costs held answers, so I hold the answer when a number cannot be found, and I tell the user exactly what to retake. And the target is p95, not the average. An average of 1.6 seconds with a p95 of 9 seconds fails this design, and the p95 is where the 4K photos live.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series — Lesson 165
GraphRAG: Retrieval Over a Knowledge Graph

This lesson retrieved text chunks and old screenshots from a flat index. The next one asks what happens when the knowledge itself is a graph, and the answer has to walk the edges between things instead of searching a pile of chunks.

Continue to Lesson 165 →