What a video-language model is, and where the action comes from
A vision-language model is two encoders and a shared vector space. Feed it video and the hard part becomes time. Ask it to act and you need a different model entirely. The papers, in order, from ImageNet to RT-2.
Every model that reads a camera feed today sits on three older results. In 2009 Fei-Fei Li and her collaborators published ImageNet (Deng, Dong, Socher, Li, Li and Fei-Fei, CVPR 2009), a labelled dataset large enough that a model could learn what things look like. In 2012 Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton trained a deep convolutional network on it, AlexNet (NIPS 2012), and won the ImageNet challenge by a margin that ended the argument about deep learning for vision. In 2017 Vaswani and colleagues published Attention Is All You Need, the Transformer, which both language and vision now share. In 2020 Dosovitskiy et al. cut images into 16 by 16 pixel patches, fed the patches to a Transformer as if they were words, and called it the Vision Transformer, or ViT. At 224 by 224 pixels that is 196 patches per image.
The joint embedding
A vision-language model, in its simplest form, is two encoders trained to agree. An image encoder f_v maps a picture x to a vector. A text encoder f_t maps a caption y to a vector of the same size. Training pushes the two vectors together for matched pairs and apart otherwise. Write s_ij = sim(f_v(x_i), f_t(y_j)) for the cosine similarity between image i and caption j in a batch of N pairs. The contrastive loss for image i is L_i = -log( exp(s_ii/τ) / Σ_j exp(s_ij/τ) ), with τ a temperature, and the symmetric term for the caption is added.
That is CLIP (Radford et al., 2021), trained on 400 million image and text pairs from the web. The result is a space where "a forklift in an aisle" and a photograph of one land close together, so you can classify an image against phrases the model never saw as labels. CLIP scores captions; generating them came later.
Getting a model to talk about a picture came next. Flamingo (Alayrac et al., DeepMind, 2022) kept a pretrained language model frozen, kept a pretrained vision encoder frozen, and trained a small set of cross-attention layers between them, so the language model could condition its next-word prediction on visual features. LLaVA (Liu, Li, Wu and Lee, 2023) went simpler: a linear projection from CLIP visual features into the token space of an open language model, trained on conversations about images that GPT-4 had written. Most open vision-language models since then follow that shape.
What changes when the input is video
A camera at 30 frames per second produces 108,000 frames an hour. At 196 patches each, that is over 21 million visual tokens for one hour of one camera, before any text. So the first decision in any video model is the sampling rate, and every choice loses something. Sample one frame a second and a 400 millisecond fall can be missed. Sample densely and the context fills in minutes.
The research has come at this from several sides. VideoMAE (Tong, Song, Wang and Wang, NeurIPS 2022) pretrains a video Transformer by masking a very high proportion of space-time tubes and reconstructing them, which works because neighbouring frames are redundant. Video-LLaMA (Zhang, Li and Bing, 2023) adds a video Q-former that compresses per-frame features into a small set of query tokens carrying temporal order, and feeds those to a language model. Video-ChatGPT (Maaz, Rasheed, Khan and Khan, 2023) built 100,000 video-instruction pairs and a benchmark for judging the answers. And Google's Gemini 1.5 technical report (2024) took the brute-force route, a context window of millions of tokens with near-perfect recall reported to at least 10 million tokens, enough to put hours of video into a single prompt.
Long context does not settle the harder problem, temporal grounding. An image question asks what. A video question asks when, for how long, and whether the thing at minute 12 is the thing at minute 40. A model has to keep identity across frames and say "the pallet was left at 14:07 and moved at 14:31" rather than "there is a pallet." Whatever the architecture, the output of a video-language model is a description, an answer, or an embedding, which is to say words and vectors.
What it cannot do
It cannot act. It has no actuator and no notion of consequence. It will tell you a person is in the forklift lane. It will not stop the forklift.
RT-2 (Brohan et al., Google DeepMind, 2023) took a vision-language model and fine-tuned it on robot trajectories, with a trick that made the whole thing possible: the robot's actions were written as text tokens. A gripper position became a short string of integers, and the language model emitted an action the way it emits a word. Web knowledge transferred: the robot could pick up the object that "could be used as a hammer" without that phrase appearing in any robot dataset. OpenVLA (Kim et al., 2024) released an open 7 billion parameter version trained on 970,000 real robot demonstrations and reported that it beat the 55 billion parameter RT-2-X by 16.5 points of absolute task success across 29 tasks. π0 (Black et al., Physical Intelligence, 2024) replaced discrete action tokens with a flow-matching head on top of a pretrained VLM, so it outputs continuous motor commands rather than quantised ones.
Two things separate a VLA from a VLM in practice. The loop is closed: observe, act, observe the result, act again, many times a second. And latency stops being optional. A model that takes three seconds to answer is fine for a search box and useless in a control loop, which is why the labs run a small fast policy at high rate and consult the large model only when the situation changes.
For agents on fixed cameras the action is usually a message or a lock rather than a gripper, so the VLM does most of the work, but the VLA lesson still holds. An agent is a model in a loop with consequences, and the loop is the part the papers about captions never had to build.