Occlusion is the problem nobody puts in the demo
A forklift, a pallet and a person, and the moment one hides another. A video agent is judged on what it does in that second, and the sales deck skips it.
The demo clip is clean, shot in good light from a wide angle with one thing happening at a time. The model draws its boxes and the room is impressed. What the clip never shows is the second forklift pulling across the frame, the pallet stacked chest-high, the worker who steps behind both. In that second the person the agent was tracking ceases to exist, and what the agent does in the next second decides whether the count, the alert, or the near-miss log can be trusted.
Why it is hard
Occlusion is geometry. A fixed camera sees a two-dimensional projection of a three-dimensional floor, and every object in front hides everything behind it. A model that classifies frame by frame sees a person, then no person, then a person again, and has no idea whether it is the same one. The failure modes stack. A lost track becomes a missed near-miss, or it becomes two people where there was one, and either way the log is wrong.
Rish Gupta's essay on cameras as digital workers calls perception the bottleneck that gates autonomy. Occlusion is where I see perception break.
What customers already know
The people buying these systems have usually been burned before. The operations-technology lead at a large food processor, planning a new plant with several hundred cameras, told Spot AI this month that detection of occluded or dirty lenses was not available out of the box and flagged it as important. He had built in-house camera systems for counting animals in the past and watched them fail for exactly this reason.
A grocery chain's asset-protection team asked a sharper version of the question during a demo of a mobile surveillance trailer: how do AI zones stay valid when a pan-tilt-zoom camera moves on a tour? Every time the camera turns, the scene it was trained to watch is a different scene.
Then there is a slower kind of occlusion. An integrator group that resells the platform wrote in an internal recap that "camera health is the real unlock," and that the system's image-quality flags turn "every refresh conversation into a data-backed one." A lens that fogs, or a mount that sags a degree or two a month, occludes a camera as surely as a pallet does, and an agent that does not notice will happily report an empty lane for a week.
What is being done about it
Three approaches show up in the engineering work. The first is memory: tracking objects across time, so a person who vanishes behind a rack is expected to reappear and gets the same identity when he does. The second is verification: Spot AI's proposer-verifier design has a second model check a candidate event before an alert fires, so a flicker of misdetection ends as a non-event and the night supervisor's phone stays quiet. The third is promptable segmentation. Spot AI's AI Labs has written about using Meta's SAM2 to detect and track custom objects without retraining, and about pairing it with vision-language models so the system can reason about an object it has only half seen.
Thermal cameras add their own wrinkle. A Spot AI engineer told a European telecom during a demo that models trained on daytime images fail when a feed switches to thermal, and need new training data.
Ask for the bad clip
My advice is simple. When a vendor shows you a demo, ask to see the frame where it loses the person, what the count did when the forklift crossed, what happens to the zones when the camera moves, and what the system says when the lens is dirty. The food processor's OT lead asked that last one because he had built his own counting systems and watched them fail; a vendor who has been inside a real building will have an answer ready, and I have not yet seen one volunteer it.