How a video system answers "show me the forklift near the dock at 3 a.m."
Retrieval over video borrows its shape from text RAG, then has to solve two problems text never had: where in the frame, and when in the stream. The research that makes it work, and the checking step that keeps false positives down.
The question in the title has four parts. An object (a forklift), a place (the dock), a time (around 3 a.m.), and a relation (near). Each is a different retrieval problem, and a system that answers it well is doing four different things and stitching the results together.
The shape comes from text
Retrieval-augmented generation was named by Lewis et al. (2020), who paired a pre-trained sequence-to-sequence model with a dense vector index of Wikipedia, so the generator could pull in passages at query time. The retriever came from the same group: Karpukhin et al. (2020) showed a simple dual-encoder beat a strong BM25 baseline by 9 to 19 points absolute on top-20 retrieval accuracy. The index underneath is typically approximate nearest-neighbour search, as in FAISS (Johnson, Douze and Jégou 2017).
Video RAG keeps this structure. Something gets embedded and indexed, a query gets embedded the same way, the nearest matches come back, and a language model writes the answer with those matches in its context. What changes is what "something" is.
Embedding video against language
The bridge is contrastive pre-training. Radford et al. (2021) trained CLIP on 400 million image-text pairs so that an image and a caption that belong together land near each other in one space, which lets a sentence query pixels. Luo et al. (2021) pushed that image model onto clips: CLIP4Clip encodes frames with CLIP, studies how to combine them into one clip vector, and reported state-of-the-art text-to-video retrieval on MSR-VTT and four other benchmarks. Bain, Nagrani, Varol and Zisserman (2021) took the other road in Frozen in Time, training a single encoder on both images and video with a curriculum that treats images as frozen single-frame videos and then widens the temporal context, on WebVid-2M, two million weakly captioned web videos released with the paper.
For a security camera, what both papers establish is that a clip embedding is cheap to compute from a few sampled frames and good enough to rank candidates. Answering "near" takes something else.
Finding the moment
Once a camera and a rough window are known, the system has to say when inside a long recording the thing happened. That is temporal grounding, sometimes called moment retrieval. Lei, Berg and Bansal (2021) built QVHighlights, over 10,000 YouTube videos annotated with natural-language queries, the moments that match, and five-point saliency scores, and proposed Moment-DETR, a transformer that predicts start and end times directly from the query and the video. Lin et al. (2023) argued in UniVTG that moment retrieval, highlight detection, and summarisation are one problem with three label formats, defined a shared formulation, and trained one model across seven datasets.
These models return a span, and a span is what the person on the night desk wants: the twelve seconds at 03:04 where a forklift crossed the dock apron, and nothing from the other fifty-nine minutes.
How long is long
Benchmarks show how much is unsolved. Mangalam, Akshulakov and Malik (2023) built EgoSchema from over 5,000 human-written multiple-choice questions on three-minute clips, built so that answering requires a long stretch of each clip. Models with billions of parameters scored under 33 percent, with chance at 20, while humans reached about 76. Fu et al. (2024) assembled Video-MME from 900 videos and 2,700 question-answer pairs, with durations from 11 seconds to an hour. An overnight shift on a dock camera is eight hours. No published system passes a full night through a video-language model and asks it to remember 3 a.m., which is why the index exists.
The pipeline
Detect: a small model on the edge runs on every frame and writes structured records, one per detection, with class, bounding box, track ID, camera, and timestamp. Embed: clip embeddings are computed every few seconds with a CLIP-style encoder. Store: the structured records go in an ordinary database indexed by camera and time; the vectors go in an approximate nearest-neighbour index. Query: a language model turns the question into a filter (camera in the dock group, timestamp between 02:30 and 03:30, class equals forklift) and an embedding for the fuzzy remainder ("near the dock door"). The filter is cheap and cuts the candidates to a few hundred clips; the embedding ranks them; a grounding model trims each to a span. Spot AI's engineering team describes its agents as built on a "Video Retrieval-Augmented Generation (RAG) System," with rule-based retrieval and text-based video search underneath (Spot AI), which matches this layout.
The mistake I see most often is putting everything in the vector index. Time and camera are exact, and searching for them with embeddings is slower and worse than a WHERE clause.
Propose, then verify
The remaining problem is false positives. A retriever tuned for recall returns pallet jacks and reflections in the dock window along with the forklift. The design that has emerged for cutting these is a cheap proposer followed by an expensive verifier that looks at the actual frames. Yu, Cho, Yadav and Bansal (2023) built SeViLA that way: a localiser picks language-relevant keyframes and an answerer reasons over them. Wang, Zhang, Zohar and Yeung-Levy (2024) went further in VideoAgent, letting a language model decide iteratively which frames it needs, and reached 54.1 percent zero-shot on EgoSchema while looking at 8.4 frames on average.
Brown et al. (2024) found that when a task has an automatic verifier, drawing more samples keeps improving coverage over four orders of magnitude, and that without one, selection by majority vote plateaus. In video, the verifier is a large vision-language model given the candidate span and a narrow yes-or-no question: is there a forklift within a few metres of the dock door in these frames. It is too slow for every candidate and cheap enough for the twenty the proposer ranked highest.
A system that answers the title question in under a second does something modest at query time. The detection happened at 3 a.m., the embedding shortly after, and the index has been waiting since. The query is a filter, a ranking, a span, and twenty quick checks.