Segment Anything, embeddings, and how an operator teaches a camera a new object
SAM turned segmentation into a prompt. SAM 2 gave it memory across frames. Put a self-supervised embedding behind them and a site manager can define a new object class from a handful of clicks, with no retraining. The design, the papers, and the limits.
For most of the last decade, teaching a camera system to see a new thing meant collecting a few thousand labelled images, training a detector, and waiting. That is why analytics products shipped with a fixed menu: person, vehicle, maybe a hard hat. Two lines of research have since made the menu editable, and they explain what "define your own object" actually does under the hood.
Segmentation as a prompt
Segment Anything (Kirillov et al., Meta AI, 2023) reframed segmentation as a promptable task. The model returns a valid mask for whatever a prompt points at: a click, a box, a rough mask, or text. The dataset they built to do it, SA-1B, holds over one billion masks on 11 million images.
The design has three parts. A heavy image encoder, a ViT pretrained with masked autoencoding, runs once per image and produces an embedding. A lightweight prompt encoder turns clicks and boxes into a few tokens. A small mask decoder combines the two and emits masks. Because the expensive part runs once, an operator can click around a frame and get a new mask each time. When a click is ambiguous (the wheel, or the whole truck?) the decoder returns three candidate masks and lets the caller choose.
SAM 2 (Ravi et al., Meta, 2024) extended the same idea to video. The addition is a streaming memory: as the model processes frames in order, it stores the embeddings and masks of past frames in a memory bank, and a memory attention module conditions the current frame on them. Click a pallet once in frame one and the model carries it forward, re-finding it after it passes behind a rack. The paper reports better accuracy on video segmentation with three times fewer interactions than prior methods, and on still images it is more accurate and six times faster than the original SAM. Spot AI has written about using SAM 2 for generalized object detection on its own cameras; the approach itself is vendor neutral.
Why the embedding is the product
A mask answers "where is the thing I clicked." To answer "is that the same kind of thing as this" you need an embedding: a function that turns an image region into a vector such that similar-looking regions land near each other. With a fixed camera, a scene becomes a set of vectors, one per object, and questions about the scene become questions about distance.
The distance almost everyone uses is cosine similarity, cos(u, v) = (u · v) / (|u| |v|), which ignores vector length and compares direction only. A value near 1 means the regions look alike to the model. One threshold, tuned once, turns the number into a decision.
Where the vectors come from matters. CLIP-style encoders learn from captions, so their space is organised around what people write about pictures. DINOv2 (Oquab et al., Meta, 2023) is trained with no text at all: a self-supervised objective on a curated image corpus, scaled to a ViT with one billion parameters. The authors report that its frozen features beat OpenCLIP on most benchmarks at both image and pixel level, and the pixel-level part is what counts here. DINOv2 features carry enough local structure that segmentation and depth work on top of them without fine-tuning, which is what you want from a backbone that will be asked about objects nobody anticipated.
Detection with an open vocabulary
The last piece is finding candidates in the first place. Classic detectors output boxes for a closed list of classes. Open-vocabulary detectors take a phrase. OWL-ViT (Minderer et al., Google, 2022) showed that a plain ViT with contrastive image-text pretraining and a detection fine-tuning stage could locate objects named in free text. Grounding DINO (Liu et al., 2023) fused a DINO-style detector with grounded pretraining and reached 52.5 AP on COCO with no COCO training data at all. Type "orange safety cone" and you get boxes.
The workflow this enables
Put the pieces together and a new object class is a few minutes of an operator's time rather than a training run. The operator types a phrase or clicks on the object in a frame. An open-vocabulary detector or SAM turns that into a box and a mask. An embedding model turns the masked region into a vector, and that vector is stored as a prototype. On live video, every candidate region is embedded the same way and compared to the prototypes by cosine similarity; above the threshold, it is that class. SAM 2 makes the first step cheap on video, because one click on one frame produces masks across hundreds of frames, and each of those is another example the prototype can average over.
Correction is part of the loop. When the system flags a wrong region, the operator marks it, and that vector becomes a negative prototype. The encoder weights never change; what changes is a small set of vectors and a threshold, which is why the cycle takes minutes and can run per site.
Where it breaks
I would not want a buyer to read this as "solved." Cosine similarity in a general embedding space measures visual likeness, and two things that look alike can mean different things on a floor. A blue tote and a blue recycling bin may sit closer together than two totes under different lighting. Prototypes drift with the seasons. The heavy encoders (a ViT-H for SAM, a billion-parameter DINOv2) do not run on a cheap edge box at full frame rate, so real deployments run them sparsely or distil them down. And the threshold that works on a well-lit dock will not work in a dim loading bay at 5 a.m.
None of that undoes the shift. In 2023 Kirillov and colleagues showed a model that could mask anything you pointed at; a year later Ravi and colleagues carried the mask through time. What the operator now supplies is a few examples and a decision about how similar is similar enough.