How a large vision model gets onto a box next to a camera

A camera on a loading dock produces a frame roughly every 33 milliseconds. The model that looks at each one usually runs on a fanless box bolted to a wall, with a power budget of 7 to 25 watts. The models that do best on public benchmarks were trained on clusters and assume a data-center card. The engineering problem is getting from one to the other without losing the accuracy that made the big model worth training. Three techniques do most of the work.

Distillation

The idea comes from Hinton, Vinyals and Dean (2015). Train a small "student" network to match the softened output distribution of a large "teacher" rather than the hard labels alone. The teacher's near-misses carry information a label does not: a frame the teacher scores as 80 percent forklift and 15 percent pallet jack tells the student something about what those two things share. The method now sits under most small vision models that ship, because it lets a small architecture inherit what a large one learned from far more data.

Quantisation

Store and compute weights in fewer bits. Jacob et al. (2017), from Google, laid out integer-only inference with 8-bit weights and activations, plus a training procedure that simulates the rounding so the network learns to tolerate it. That scheme is why edge accelerators are marketed in INT8 TOPS rather than floating-point FLOPS.

Transformers turned out to be harder. Dettmers et al. (2022) showed that large models develop outlier features that break naive 8-bit conversion; their LLM.int8() handled those few dimensions in higher precision while multiplying more than 99.9 percent of values in 8-bit, halving memory with no measured degradation up to 175 billion parameters. Frantar et al. (2022) went further with GPTQ, a post-training method that takes weights to 3 or 4 bits and quantised a 175-billion-parameter model in about four GPU hours. Lin et al. (2023) observed in AWQ that protecting roughly 1 percent of salient weights, chosen by looking at activations rather than the weights themselves, removes most of the quantisation error, and reported more than a 3x speedup over FP16 on desktop and mobile GPUs. These papers were about language models, but the vision-language models now being pushed onto edge boxes carry the same transformer blocks, and the same tooling applies.

Pruning

Remove weights that contribute little. Han, Pool, Tran and Dally (2015) trained, pruned, and retrained, and cut AlexNet from 61 million parameters to 6.7 million with no loss of accuracy. Frankle and Carbin (2018) argued that dense networks contain sparse subnetworks, at 10 to 20 percent of the original size, that train to the same accuracy from their original initialisation. Scattered zeros only speed things up if the silicon can skip them, so most edge deployments prune whole channels instead, for smaller gains that show up in measured latency.

Architectures built small from the start

Howard et al. (2017) built MobileNets on depthwise separable convolutions, with width and resolution multipliers that let an engineer pick a point on the accuracy-latency curve. Tan and Le (2019) scaled depth, width, and resolution together in EfficientNet; their B7 model reached 84.3 percent top-1 on ImageNet while being 8.4x smaller and 6.1x faster at inference than the best existing convolutional network.

The TinyML line took this to its extreme. Lin et al. (2020) reported over 70 percent ImageNet top-1 accuracy on an off-the-shelf commercial microcontroller with MCUNet, under a memory budget measured in kilobytes. A camera analytics stack is unlikely to run on a microcontroller, but the discipline carries over: on an edge box, SRAM and memory bandwidth constrain you before compute does, and a model chosen by FLOP count alone will often disappoint.

Read it as a curve

Every one of these techniques is a move on the same chart: accuracy on the vertical axis, latency on the target hardware on the horizontal. Quantisation and pruning slide a point left, usually with a small drop. Distillation and a better base architecture raise the curve itself. I have watched teams pick a model from a leaderboard and then find that the curve on their own box looks nothing like the paper's, because a rated TOPS figure says little about per-frame latency once memory traffic is counted. The curve has to be measured on the box that ships.

NVIDIA's Jetson Orin family runs from the Orin Nano 4GB, rated at 34 TOPS between 7 and 25 watts, to the AGX Orin at 275 TOPS between 15 and 60 watts (NVIDIA). When Qualcomm Ventures invested in Spot AI in 2024, its senior director Tushar Gupta said the deal "closely aligns with Qualcomm's vision of implementing AI on edge to solve real-world challenges."

Why the design has two tiers

The reason the small model on the box does not have to do everything is a latency budget. Rish Gupta's essay on cameras as digital workers writes it as a sum: the time from perception to action is the time to capture a frame, plus encode, plus infer, plus decide, plus actuate, plus whatever the network adds. His figure for most security and safety work is one to two seconds; for robotics it is 50 to 200 milliseconds.

The sum sorts the work. Detection and tracking have to happen inside that budget on every frame, so they run on the distilled, quantised model on the box, where the network term is close to zero. Reasoning about what a detection means, such as comparing it against a written policy, can wait several seconds, so it goes to a large model in the cloud on a clip or a metadata record rather than a raw stream. The two-tier layout is standard across the category because the arithmetic leaves little room for anything else.

For a buyer this reduces to a question for the vendor: which model runs on the box, what was it distilled from, and at what bit width. The answers say more about what the system can catch at 3 a.m. than a demo does.