Occlusion is the problem nobody puts in the demo
A forklift, a pallet, a person. The moment one hides another is the moment a video agent is judged. Why the hardest problem in physical AI is the one the sales deck skips.
The demo clip is always clean. Good light, a wide angle, one thing happening at a time. The model draws its boxes and the room is impressed. What the clip never shows is the second forklift pulling across the frame, the pallet stacked chest-high, the worker who steps behind both. In that second the person the agent was tracking ceases to exist, and what the agent does next is the whole product.
Why it is hard
Occlusion is not a bug. It is geometry. A fixed camera sees a two-dimensional projection of a three-dimensional floor, and every object in front hides everything behind it. A model that classifies frame by frame sees a person, then no person, then a person again, and has no idea whether it is the same one. The failure modes stack: a lost track becomes a missed near-miss, or a double count, or an alert that fires twice for one event.
Rish Gupta's essay on cameras as digital workers puts the general point bluntly: perception is the bottleneck that gates autonomy. Occlusion is where perception most often breaks.
What customers already know
The people buying these systems have usually been burned before. The operations-technology lead at a large food processor, planning a new plant with several hundred cameras, told Spot AI this month that detection of occluded or dirty lenses was not available out of the box and flagged it as important. He had built in-house camera systems for counting animals in the past and watched them fail for exactly this reason.
A grocery chain's asset-protection team asked a sharper version of the question during a demo of a mobile surveillance trailer: how do AI zones stay valid when a pan-tilt-zoom camera moves on a tour? Every time the camera turns, the scene it was trained to watch is a different scene.
Then there is the slow occlusion nobody names. An integrator group that resells the platform wrote in an internal recap that "camera health is the real unlock," and that the system's image-quality flags turn "every refresh conversation into a data-backed one." A lens that fogs, a mount that sags two degrees a month, a spider web across the dome at dawn: these occlude a camera as surely as a pallet, and an agent that does not notice will happily report an empty lane for a week.
What is being done about it
Three approaches show up in the engineering work. The first is memory: tracking objects across time so a person who vanishes behind a rack is expected to reappear, rather than being born again as a stranger. The second is verification: Spot AI's proposer-verifier design has a second model check a candidate event before an alert fires, which turns a flicker of misdetection into a non-event instead of a page to the night supervisor. The third is promptable segmentation. Spot AI's AI Labs has written about using Meta's SAM2 to detect and track custom objects without retraining, and about pairing it with vision-language models so the system can reason about an object it has only half seen.
Thermal cameras add their own wrinkle. A Spot AI engineer told a European telecom during a demo that models trained on daytime images fail when a feed switches to thermal, and need new training data. Night is an occluder too.
Ask for the bad clip
The practical advice is simple. When a vendor shows you a demo, ask to see the frame where it loses the person. Ask what the count did when the forklift crossed. Ask what the system says when the lens is dirty. A team that has an answer has been in a real building. A team that changes the subject has been in a demo.