Four benchmarks for an agent that has to act
Describing a scene is a solved demo. Doing the right thing in it every hour of a shift is the harder test, and buyers have started writing four metrics for it into their evaluations.
The demo ends with the model looking at a clip and saying what is in it. A forklift, say, or a door someone left open. The audience nods. I have watched that nod in a dozen rooms and it has never once been a purchase order, because the next step is a building with forty cameras and a night shift, and there the question becomes whether the thing can be trusted to act, every hour, without wearing the people on that shift out.
The benchmarks that answer that question are different from the ones that win the demo. Four of them are showing up in real evaluations.
1. Alerts a human would thank you for
Count the alerts that deserved a person's attention and divide by the ones that did not. That ratio is the number. One retailer's internal review, shared at a Spot AI planning session this summer, put the problem in a single figure: more than 65,000 alerts, and not enough asset-protection staff to process them. Spot AI's own first newsletter conceded that "over 90% of security alerts are harmless: think errant shopping carts or idling delivery trucks."
An advisor with a long career in remote guarding put the metric more precisely in a call with the company: measure mean time to first action, and the ratio of valid dispatches to customer complaints. Raw alarm counts, he argued, tell you nothing.
2. Latency you can act inside
Spot AI advertises under one second from event to alert for its security agent. The reason that number is on the marketing page is that buyers ask for it. A partner integrating the platform into its own dispatch workflow reported that a clip fetch taking around 82 seconds was too slow for real-time response, and asked for a handful of still frames delivered at once. The general version of that request is the time until the person who has to do something has what they need. Model inference is one slice of it.
3. Governance of the errors
The most telling document in this category is a manufacturer's evaluation checklist. A global animal-health company sent Spot AI ten numbered requirements before agreeing to a demo. Three of them were about what happens when the model is wrong:
- "False positive governance — How errors are measured, tuned, and managed over time"
- "Human-in-the-loop — How low-confidence or high-risk exceptions are routed to human reviewers"
- "Auditability — Whether alerts can be traced to source frame, timestamp, model version, reviewer, and disposition"
There is no computer-vision benchmark in that list. The three items are what a compliance reviewer will ask about. Spot AI's engineers describe a proposer-verifier design in which a second model checks each candidate event before it becomes an alert, and the company has discussed letting partners choose whether they would rather tolerate more false positives or more false negatives. That choice is a benchmark too: the right error rate depends on what the alert triggers.
4. The outcome the budget owner recognises
The last benchmark is the only one a finance team reads: a loss line that moved. Spot AI's published customer results are framed this way for a reason. A grocery chain with 27 percent fewer incidents across twelve stores, an apparel retailer whose cash shrink fell from 6 percent to 1 percent, a manufacturer reporting a 15 percent improvement in safety compliance. Whatever you think of vendor-reported numbers, the shape is the right one, because each starts from a figure the finance team was already tracking before the pilot.
What to ask for
Put the four together and you have a scorecard a buyer can run in an eight-week pilot: the share of alerts that earned attention, the time to a usable alert, the audit trail behind each one, and a baseline loss figure taken before day one. Vision accuracy gets a vendor into the pilot. Whether the system is still installed in week nine comes down to those four numbers, and I would ask for them in writing before the first camera is connected.