Some models live in a data center. This one lives in a handheld scanner at the gate of a depot. A worker points it at a parcel label, and a small model on the device reads the postcode and picks the bin the parcel goes in. No network call. No waiting for a server. The photo never leaves the scanner. That is on-device AI, and it is a different job from serving a model behind an API. The model has to be small enough to fit, fast enough to keep up with a worker, and tough enough to run where the signal does not reach. This lesson follows one scan through nine hops, from the tap on the button to the report that syncs later, when the signal comes back. It also shows the two other ways that trip can go: a shed with no signal at all, and a model update that ships the wrong file and slows every scan down. I learned this the same way I learn every system: by watching what happens when the network goes away, and by measuring what a model costs on the weakest device in the fleet, not the newest one. The formats have names you will meet everywhere. TFLite is the small runtime for phones and handhelds. ONNX is the format that carries a model between frameworks and runtimes. Core ML is the format for one maker's phones. Lesson 87 taught quantization, the trick that shrinks the model so it fits. Lesson 57 taught inference, the trip a request takes through a model. This lesson puts both of those ideas into a device you can hold in your hand.
Every hop charges rent in milliseconds. The bar below is a worked example for one scan, not a measurement from a live system, but the shape is honest: the sync is the biggest slice, and the worker never waits for it. In the normal trip the on-device hops add up to 245 ms and the sync adds 180 ms. Offline, the trip is 245 ms and the sync waits in a queue. The bad update costs 952 ms, because the model is four times bigger and runs on the wrong processor. Step through the lesson or switch the scenario and watch the bar change.
The budget above is one fixed trip. This box is yours. Change the model size, change the quantization, change how many scans the device does in a minute, and watch the trade move: a smaller file loads faster and runs faster, a coarser quantization costs accuracy, and more scans an hour costs battery. The on-device trip here does not count the sync. The sync is background work, and the worker never waits for it.
The size of the model before quantization, in megabytes. A bigger model knows more postcodes and more handwriting styles, and it costs more of everything: file size, load time, run time, battery.
How many bits each weight gets. 32-bit is the model as trained. 8-bit is the usual on-device trade: a quarter of the size, about 2 points of accuracy lost on our labels. 4-bit is smaller still, and it costs 3 to 6 points of accuracy. Test it on your own labels before you ship it.
How many parcels the worker scans every minute at peak. The per-scan cost stays the same, but the battery burn and the queue grow with the rate, and the rate decides whether one device keeps up.
Try this: (1) Set quantization to 32-bit at the defaults. The file jumps from 30 to 120 MB, load from 60 to 240 ms, inference from 85 to 301 ms. (2) Now drop to 4-bit. The file is 15 MB and inference is 49 ms, but expect 3 to 6 points of accuracy loss, so test it before you ship it. (3) Push scans to 120 a minute. The device can do about 244 scans a minute at the defaults, so 120 fits, but the battery burn climbs to about 52 percent of a 4,000 mAh battery per hour. The per-scan cost is fixed. The battery bill is not.
Three questions. No score is saved anywhere. This is just you against the material.
I read this topic four times here. First as one scan on one device, then as a quantized file in a format, then as a fleet that updates and reports back, then as a system that has to keep working when the network does not.
Definition: On-device AI means the model runs on the device that collects the data, not on a server. The input is a photo, a sound, or a sensor reading. The output is an answer the person holding the device can act on, right now, with no network.
Types:
Uses: I use the basic level for one concrete job. A worker scans a parcel, the model reads the postcode, the screen shows the bin. The whole trip is 245 ms and none of it needs the network. The sync to the server happens later, when the signal comes back.
Dimensions: At low load on-device AI feels free, because the device is already in your hand and the server bill is zero for the answer itself. The cost shows up in the build: a smaller model, a conversion step, and a test on the real device.
Do this: Time one scan on paper. Tap, photo, answer. If the whole trip is under a second, the device did the job. If it needs a loading spinner, the network did.
Definition: A model trained in 32-bit floats is too big for a device. Quantization rewrites the weights in fewer bits, usually 8, so the file is a quarter of the size. The formats are the shipping boxes: TFLite for small runtimes, ONNX for moving between frameworks, Core ML for one maker's phones.
What changes: The file gets smaller, the load gets faster, the inference gets faster, and the accuracy drops a little. How much it drops is a measurement, not a guess. I measure it on the labels the device will actually see, with the preprocessing the device actually runs.
What I do: I convert the model in the build, not by hand. The converted file gets its own version number, its own accuracy test, and its own run on the slowest device in the fleet. A converted file I never tested on a device is a guess with a version number.
Do this: Take the 120 MB number from this lesson. At 8-bit it is 30 MB. At 4-bit it is 15 MB. Work out both on paper, then check them against the simulator above. The numbers should match.
Definition: A fleet of devices is a distributed system. The model ships like software: a manifest with a version, a file, and a checksum, a staged rollout, and a rollback that is one manifest change. Telemetry comes back per scan: version, inference time, confidence, how often the app asked the worker.
What changes: The model stops being a file and becomes a release. A bad version reaches 5 percent of the fleet first, and the telemetry tells me before it reaches the rest. The confidence threshold becomes a product decision: below the line, the app asks a human instead of guessing.
What I do: I alert on inference time and on the low-confidence rate, split by model version. A new version that is slower or less sure than the old one pages someone. And I keep the queue bounded, because a device that spent a day offline must not fill its storage with unsent reports.
Do this: Write down three numbers for your own model version: inference p95, low-confidence rate, fallback rate. If you cannot fill in one of them, that is the telemetry you are missing.
Definition: At fleet scale the questions are bandwidth, battery, and trust. Ten thousand devices downloading a 120 MB update is a bandwidth bill and a slow rollout. A model that burns 13 percent of the battery an hour changes how long a shift lasts. A model that guesses wrong on one device in a thousand is a hundred wrong parcels a day.
What changes: Updates go out in waves, with a kill switch. The model registry holds every version with its accuracy score and its size. The dashboard shows per-version inference time, confidence, and fallback rate, and a version that drifts gets pulled. The server keeps the jobs that do not fit on a device: retraining, the big model, and the fleet-wide counts.
What I do: I decide what runs where with a rule, not a feeling. Under 200 ms, private data, or no signal means the device. Retraining, big models, and fleet answers mean the server. And I test the whole thing on the oldest device a worker still carries, because that is the device the design has to serve.
Do this: Pick your fleet size and multiply. 1,000 devices times a 30 MB update is 30 GB of downloads. 10,000 devices is 300 GB. That number decides whether your rollout goes in waves.
The lesson above is one scan through nine hops. Production is the rest of it. Here is what I actually spend my time on once the model runs on a device and the budget is written down.
I decide with a rule, not a feeling. The device gets the job when the answer has to come back in under about 200 ms, when the input is private and should never leave the device, or when the work has to happen with no signal. The photo of a parcel label, the voice command in a living room, the vibration reading on a factory floor: all three belong on the device. The server keeps the jobs that do not fit: retraining the model, running the big model that is too large for a phone, and answering fleet-wide questions like how many parcels went to each town today. The sync hop in this lesson is the boundary between the two. The answer is local. The counting is central. When I am unsure, I ask one question: does the user need this answer before the network can answer it? If yes, it runs on the device.
TFLite is the small runtime for phones, handhelds, and single-board computers. You convert a trained model into a .tflite file, the runtime loads it, and the chip or the processor runs it. It is the format I reach for when the device is small and the battery matters. ONNX is the format that carries a model between frameworks and runtimes. You export to .onnx once, and ONNX Runtime runs it on a laptop, a server, or a device, on CPU or on an accelerator. It is the format I reach for when the same model has to run in more than one place. Core ML is the format for one maker's phones and tablets. You convert to .mlpackage, the system loads it, and the neural engine in the chip runs it fast and cool. It is the format I reach for when the whole fleet is one maker hardware. The lesson is the same in all three: conversion is a build step, and the converted file is what you test, not the model you trained.
I ran this on my own machine, so the numbers below are measured, not guessed. First, the numpy version of int8 quantization, the same math a converter does to the weights. One thousand weights, 32-bit floats, squeezed to 8-bit integers and back.
Output from the run: fp32 bytes 4000, int8 bytes 1000, a ratio of 4.0. Scale 0.009709. Max absolute error 0.00485, mean absolute error 0.00244. A 120 MB model becomes 30 MB. A 30 MB model becomes 7.5 MB. The error is small because the weights are small numbers and the scale is small. The error is not zero, and that is the whole trade: you pay a little accuracy for a quarter of the size.
Then I built a tiny two-layer model in ONNX, a stand-in for the parcel sort model, and ran it with ONNX Runtime 1.32.0, 2,000 runs each. The fp32 model file was 902 bytes. The int8-weights version was also 902 bytes in this toy, because I stored the dequantized weights as floats, so the file did not shrink. That is honest to say: my toy did not prove a size win, it proved the run works. Mean time was 0.031 ms per run for fp32 and 0.043 ms per run for the int8-weights version. On a toy this small the quantized version was not faster, because the dequantization overhead is bigger than the math it saves. The speed win shows up at real model sizes, where the weights are the bottleneck. I say that plainly because the lesson would be wrong if it claimed quantization is always faster. It is always smaller. Faster depends on the size.
One honest note from the same machine. I tried to install tflite-runtime in the VM, and pip reported no matching distribution for this platform, so I could not run a TFLite interpreter here. The TFLite format is still the right call for a handheld, and the conversion step belongs in the build. I just did not pretend to run it.
A model on a device is software, so it ships like software. The server holds a manifest, and the device asks for it when it has signal. The manifest names the version, the file, the checksum, and how much of the fleet should get it. The device downloads the file, checks the checksum, and swaps it in on the next cold start. The old file stays on the device until the new one proves itself, because the rollback has to work with no signal too.
The rollout goes in waves: 5 percent, then 25, then 100. Each wave waits for telemetry before the next one starts. The gates are the same numbers this lesson watches: inference time p95, low-confidence rate, and how often the app asks the worker. If a wave trips a gate, the manifest flips rollout_percent back to 0 and rollback_to becomes the live version. The devices pick that up on their next check, with no code deploy and no app store wait.
Every scan sends a small report: model version, inference time in ms, confidence, whether the app asked the worker, and the queue depth at send time. That is enough to watch the model. It is not enough to reconstruct a parcel, and that is on purpose. The photo never leaves the device. The postcode never leaves the device as a field you can search. The server counts parcels per bin, not parcels per person. I sample the reports at full rate when the fleet is small, and I sample one scan in ten when the fleet is large, because the bill for telemetry grows with the fleet and the shape of the data does not change. And I keep a per-version dashboard, because an average across versions hides the one version that is broken.
At 10x the scans, the battery is the first thing that breaks. A scanner doing 30 scans a minute at 145 ms of chip work per scan burns about 13 percent of a 4,000 mAh battery an hour. At 120 scans a minute it burns about 52 percent an hour, and the shift ends early. The fix is a smaller model or fewer scans per charge, not a bigger battery. At 10x the devices, the update bandwidth breaks. Ten thousand devices downloading a 120 MB file is 1.2 TB of downloads, and a staged rollout turns that into a week of waves instead of a day. At 100x, the model registry becomes the system. Every version has a record: its training data, its accuracy, its size, its rollout state. The per-version dashboard is the on-call surface, and the kill switch is a manifest change. The device still does the same nine hops. The fleet around it is what grew.
This one stayed with me because the model was fine on every dashboard the team watched, and the workers had already stopped using it.
A depot fleet of handheld scanners got a model update on a Monday morning. By midday the workers had gone back to typing postcodes by hand. The scans were slow, the model was unsure, and the app kept asking the worker to pick the bin. Nobody had paged, because the server was healthy and the sync was working. The model was the only thing that had changed, and the model had no alert on it. The workers did not file a ticket. They just worked around it, and the parcels kept moving, slower, by hand.
Four things. First, the build now checks the output file size and fails the pipeline if the converted file is bigger than the last release by more than 10 percent. A missing conversion step can no longer ship silently. Second, model updates roll out in waves, 5 percent first, and the wave waits for telemetry before the next one starts. Third, inference time p95 and the low-confidence rate have alerts, split by model version, so a bad version pages someone instead of waiting for a worker to complain. Fourth, the old model file stays on the device until the new one proves itself, so a rollback works even on a device with no signal.
This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real device. Take a minute with it before you read my approach.
Design on-device defect spotting for a factory line. A sensor gateway on the line photographs parts on a belt and flags the defective ones. The line moves at 50 parts a minute. The factory floor has patchy signal. A missed defect costs a customer return. A false flag stops the line for a human check. The gateway runs on wall power, but the camera and the chip share a small enclosure that gets hot. Sketch what runs on the device, what stays on the server, and the budget.
I would ask five things before I sized anything. What does a defect look like, and how many labelled photos of good and bad parts exist? How long can one part wait for an answer before the belt carries it past the reject arm? What happens when the model is unsure: does the part go to a human, or does the line stop? Where do the photos go, and can a photo of a part leave the factory at all? And what is the failure we design for: a bad model update, a hot enclosure that throttles the chip, or a day with no signal?
Start with the budget from this lesson. 50 parts a minute is one part every 1,200 ms, so the on-device trip has 1,200 ms per part. I spend it like this. Capture 40 ms, preprocess 25 ms, load 0 ms because the model stays loaded on wall power, inference 85 ms with the 8-bit model, postprocess 12 ms, decide 8 ms, show the flag 15 ms. That is 185 ms of trip, with 1,015 ms of headroom. The headroom is the point. It absorbs a hot enclosure that throttles the chip to half speed, which would push inference to about 170 ms and the trip to 270 ms, still inside the budget. The model is a 30 MB 8-bit file, and the gateway holds two versions so a rollback works offline. The defect photos stay on the gateway. Only the counts go to the server: parts seen, parts flagged, confidence, model version. The server retrains weekly on the flagged photos a human reviewed, and the new model ships in waves.
A bigger model buys accuracy and costs heat and time in a small enclosure, so I start with the smallest model that passes the labelled test and grow it only when the misses say so. A low confidence threshold buys fewer misses and costs more false flags, and every false flag stops the line, so I set the threshold from the cost of a stop, not from a round number. Running the model on the gateway buys an answer with no signal and costs a conversion and test step per release, so I convert in the build and test the converted file. Sending photos to the server buys faster retraining and costs privacy and bandwidth, so I send counts always and photos only when a human reviewed them on site. And the 1,200 ms budget is per part, not average. An average of 185 ms with a p99 of 1,500 ms fails this exercise, because the slowest part is the one that misses the reject arm.
Saved in this browser, so the Lessons index can show your progress across the course.