From composable models to reasoning: how video agents got built
Spot AI stacked small detection models together until the stack ran out of judgement. A large multimodal model closed the gap, and the agent category came out of the seam between the two.
Spot AI's AI engineering team told its origin story on the Google Cloud blog in November 2024, and it starts with a stack of small models on a box next to the cameras. It ends with a large model, because the small ones hit a ceiling.
The post, "Engineering at Speed: Combining Edge AI with the Latest Cloud Intelligence to Build Video AI Agents," is credited to Nate Lee and Dunchadhn Lyons of Spot AI's AI engineering group. It opens with what the system is meant to catch:
A forklift turns a corner too quickly. A worker forgets their safety gear. A workstation sits empty during peak hours. These moments matter.
Building blocks first
The first approach was composition. An internal positioning document from August 2024, written ahead of that autumn's agent launch, spells out the recipe: a forklift model plus a people model plus a set of drawn zones equals a forklift near-miss model. Each block is a narrow detector that learns one thing well. Combine the outputs and add rules about distance and speed, and you have a detector no single model was trained for.
The same document explains why the company started with small models. "Large models understand a lot of things reasonably well; video AI winners build models that learn a few things really well." It also records the mix of work at the time, roughly 80 percent composed agentic models, 10 percent large-model object detection and 10 percent semantic search.
The engineering post describes composable edge models for object interactions, speeds and spatial relationships, and its verdict is that they got the team "part way there."
Where the stack ran out
The ceiling was context. In the post's words:
Traditional computer vision systems excel at identifying objects but struggle to comprehend context and relationships – like someone walking dangerously close to a forklift.
A forklift detector and a person detector can tell you both are in the frame. Neither can tell you whether the person is a spotter doing their job or a visitor who wandered in, and a rough answer at scale means alerts the team learns to ignore.
The fix described in the post was a reasoning step from a large multimodal model. Spot AI integrated Gemini's video API on Google Cloud and used it to verify what the edge detectors proposed. The edge finds candidates cheaply and quickly; the large model looks at the clip and answers the harder question, whether this is the event anyone cares about. The post calls the combined design the "first Video Retrieval-Augmented Generation System designed specifically for real-time physical world understanding."
The August 2024 document had anticipated this with a one-liner, "LLM is old news, new news is foundation models for video." The enabling shift, in its argument, was GPU compute at the edge plus foundation models that could take video as input.
Why the reasoning step made it an agent
Detection plus verification is still a better alarm. What made the company call the result an agent was a third step:
We've created a system that doesn't just see what's happening in your environment – it understands the context and can take automatic action to drive outcomes.
The internal document says the same thing in fewer words: "Agents = outcome generators." Verification is what makes the action safe to automate. If the model that fires the alert would also fire on every passing shadow, no plant manager will let it lock a door. Once a second, more capable model has checked the clip, some will.
The post lists what customers reported once the loop was closed: manufacturing safety directors reducing injuries by 40 percent, auto-services owners quadrupling membership rates, school principals cutting incident-resolution time by 80 percent. Cornel Stewart, an engineer at the industrial foam manufacturer Elite Comfort Solutions, is quoted by name:
Spot AI led us from having a video surveillance system to fully leveraging video AI to drive business results. Not only have we seen a reduction in injuries and incidents, but the system's ability to identify potential hazards has allowed us to address issues before they escalate.
I find the sequence more instructive than the results. The team built the cheap layer first, found its limit in context, and bought context from a large model only where it was needed. Edge first, cloud for judgement, is now how most of the category is built. The market the agents walked into is the one the August 2024 document described: "The Video Surveillance market - all products and business models - have been built ground-up for humans watching video."