What “in-context learning” means for a robot
A new paper on GPT-Policy (Cheng, Yi, Fang, et al., from Morphi Robot and collaborators) asks a more interesting question: what if a robot could just learn from what’s in front of it, the same way people do?
In language models, in-context learning is the ability to pick up a new pattern from examples given in the prompt, without updating any weights. The GPT-Policy authors extend this idea to physical robots: can a general-purpose vision-language model (VLM) - something like GPT-6 Astra - watch a demonstration, read a goal image, or remember what it just tried, and turn that into correct robot actions on the spot?
They formalize this as five distinct kinds of context a robot might receive at test time:
- Human video - watching a person do the task
- Robot video (with or without recorded actions) - watching a teleoperated demonstration, sometimes paired with the exact end-effector poses and gripper states
- Goal image - a picture of what the finished arrangement should look like
- Self-interaction history - the robot’s own recent observations and outcomes within the same episode
- Online human-robot interaction - live cues like gestures or turn-taking in a game
No gradient updates. No fine-tuning. The model’s parameters stay frozen; all the adaptation happens through what’s fed into the prompt.
Why this matters for dynamic environments
Static training data can’t anticipate every arrangement, every lighting condition, or every new object a robot will encounter. That’s exactly the kind of variability that shows up in unstructured or high-mix manufacturing, warehouses, homes - anywhere the environment isn’t perfectly controlled. If a robot could absorb new information from a single demonstration or a single reference image at deployment time, the cost of adapting to a new situation drops from “retrain a model” to “show it once.”
The architecture that makes this possible is fairly elegant. A context compiler takes videos, images, or interaction history and turns them into a structured sequence of labeled images and text the VLM can reason over. The VLM then proposes a robot action - a tool call, essentially, like “move to this pose” or “close the gripper.” A constrained execution layer checks that the action is physically valid (inverse kinematics, joint limits, collision-aware sampling), executes it, and reports back what actually happened. That feedback loops into the next decision. The VLM never controls the robot directly; it only ever expresses intent through a verified interface.
What the real-robot trials showed
The team ran real hardware trials across all five context types, and the pattern was consistent: context helped, sometimes dramatically.
- Human video, with zero robot action labels, took task success from 0/3 to 2/3 on two manipulation tasks, while also cutting the number of decisions and execution time by roughly 20–30%.
- Robot video + recorded actions pushed a contact-sensitive bottle-unscrewing task to 3/3 success, versus 0/3 with no demonstration and 2/3 with video alone - the numerical action trace apparently resolved ambiguity that pure video keyframes left underspecified.
- Goal images let the model hit 3/3 on spatial arrangement tasks, conveying position and spacing information that’s awkward to spell out in text.
- Self-history let the robot reason across an entire episode - in one case, the model figured out on its own that it needed to move a towel out of the way to find a hidden plate, purely from remembering its own prior observations.
- Live human interaction allowed a robot to play a legal, competitive game of tic-tac-toe and track turn-taking, purely from context accumulated during the interaction.
Comparisons across models (GPT-6 Astra, Fable 5.1, Kimi K3) also suggested this isn’t a universal capability - some models made much better use of the same demonstration video than others, with GPT-6 Astra reaching full task progress where the other two stalled partway through.
The gap that remains
The paper is refreshingly honest about where this breaks down. Understanding what to do and reliably doing it are different problems. The authors repeatedly observed things like inter-arm collisions during bimanual tasks - the model had the right idea but no dedicated safety layer was watching both arms’ planned paths together. Good context can guide task-level reasoning while still leaving contact precision, outcome verification, and physical safety unresolved. There’s also a real cost dimension: each VLM decision is comparatively slow and token-expensive next to a specialized low-level controller, which the authors flag as a reason to eventually pair a deliberative model like this with a faster, cheaper controller for fine motor execution - a “System 1 / System 2” split.
The takeaway
The headline result is almost understated: a little context goes a long way. Off-the-shelf, general-purpose VLMs - never trained specifically as robot policies - can already use demonstrations, goal images, and interaction history to adapt behavior at deployment time, without touching a single weight.
That’s a meaningfully different path toward flexible automation than “collect thousands of labeled examples per object.” The frontier now isn’t whether general models can understand a new situation - it’s making that understanding translate into precise, safe, physically reliable execution every time.
