Describing an event in a sentence and letting the camera look for it
Most of what customers want to detect has no off-the-shelf model. Our first granted patent covers the workaround: put frames and sentences in the same vector space and treat a description as a trigger.
One site wanted to know when someone in a hoodie was loitering near an entrance. An office wanted a heads-up when a parking enforcement vehicle appeared on the street outside. Someone else wanted to know when a trash can had been left open. Our detectors, like everyone's, come as a fixed menu (people, vehicles, motion in a zone) and none of those requests is on it. None of them justify collecting thousands of labelled frames and training a detector either, and most of our customers do not have a data scientist to do it anyway.
US 12,456,299, granted on October 28, 2025, describes how we handle this. What follows is what the public filing says, and nothing that is not in it.
One space for pictures and sentences
The core idea is old in research and useful in practice. Train a model on pairs of images and their captions so that an image and a sentence describing it land close together in one embedding space, and unrelated pairs land far apart. Once you have that space, a sentence can do the job of a detector.
On the recorder, an image encoder runs over each frame and over crops of the objects our detection models have already found: people, vehicles, forklifts. Each crop gets an embedding, and the embedding is attached to the same detection metadata that already holds the bounding box, the object class and the attributes we track. Cropping matters. An embedding of a whole warehouse frame says "warehouse". An embedding of the person crop says "person wearing a helmet", which is what a safety trigger cares about. The embeddings are indexed in the cloud with an approximate nearest-neighbour structure so they can be searched later.
A semantic trigger is a description of the event encoded into the same space. At every timestamp where we have an embedding, we compute the distance between the two vectors. If the distance is under the trigger's threshold, the trigger fires. A semantic trigger composes with every other trigger in the system, so "person wearing a hoodie" can be combined with a zone, a count and a duration exactly as a presence trigger would be.
When a sentence is not enough
Some events are too specific for a sentence, or the sentence keeps matching the wrong thing. For those, the filing describes a few-shot path that a non-engineer can run.
The user gathers a small set of images that show the event and a set that does not, and writes a handful of plain-language descriptions for each class. The negative descriptions are deliberately far away: roses in a garden, a dog in a park. Because the image set is small, we augment it with variations such as lens flare and greyscale, then embed every image and every sentence. The classifier is linear and trained on the embeddings, with cross-validation, so it never touches pixels, and the result is a single weight vector in the same space. That vector, with a distance threshold, becomes the trigger. The trigger editor accepts it as an attribute type next to count and zone.
Two details in the filing were not obvious to us at the start. First, adding the text descriptions to the training set produced a clear improvement in accuracy over images alone. The sentences act as anchors that pull the image embeddings toward the concept we mean. Second, we plot the embeddings in two dimensions before training. When the two classes separate cleanly the classifier will work. When one lonely image sits on the wrong side, it usually means the concept is under-specified (a hoodie versus a jacket, say), and what fixes it is more examples of that case. More training does not.
A second opinion on the clip
Semantic similarity is a score, and scores are wrong sometimes. The filing describes a verification step that runs after a trigger fires. The clip around the event is sent to a language model that accepts video, with a prompt that describes the clip and asks a direct question: did this event occur, yes or no, followed by a brief explanation. The answer gates the action. We think of it as an ensemble of three different kinds of reasoning, the object detector, the semantic model and a model that can talk about what it saw. The explanation is kept, because a customer who gets an alert wants to know why.
Users can also correct us. A thumbs-down on an event, a redrawn zone or an uploaded example becomes ground truth that refines the trigger for that camera.
What did not work
Training on images alone gave weaker separation between classes than images with text anchors. Small image sets without augmentation did the same. Whole-frame embeddings answered object-level questions badly, which is why the crops exist. Each of these is written into the filing as a reason the method looks the way it does, and the combination that held up was a frozen encoder, object crops, text descriptions for each class and a linear head on top.
What shipped
The approach is in the product as the semantic trigger type. The patent was granted on October 28, 2025, with Nathaniel Lee, Dunchadhn Lyons, Julie Noble, Cameron Akhavan, Jordan Hart, Amrish Kapoor and Kevin Tajeran named as inventors. The independent claims cover training the shared model, embedding a plain-language description of an event, embedding frames together with their detection metadata, and firing the event when the two embeddings are close enough.