inferwire
/
AI·2 min read

RV-ICL Lets Robot Agents Learn From One Video, Clip by Clip

Instead of stuffing a demonstration video into the prompt, RV-ICL turns it into a hierarchy a robot agent can browse. With one demo per task, success rose from 86.7 to 95.8 percent on LIBERO-Plus.

TL;DR

  • RV-ICL turns a single demonstration video into a hierarchy of keyframes, phases, moments and short clips that a robot-control agent can look up as needed [^1].
  • Built on the RPent framework, it raised success from 92.6 to 96.5 percent on LIBERO-PRO and from 86.7 to 95.8 percent on LIBERO-Plus, without training [^1].

Background

Some robot systems use a language-model agent to orchestrate frozen vision-language-action (VLA) policies, which handle contact-rich motions the agent cannot perform itself [^1]. These agents improve across episodes through text memory, but text records what the agent did, not how a task is done [^1]. A demonstration video shows the how, yet fits poorly into an agent's context: the full video slows every step, and a few fixed keyframes lose the contact details that decide whether a grasp holds [^1].

What happened

Researchers from the University of Central Florida and the University of Southern California introduce Recursive Video In-Context Learning (RV-ICL), a training-free method that turns a demonstration into a hierarchy the agent navigates instead of a prompt it receives [^1]. The hierarchy is built around the demonstration's sub-events, such as grasps and releases, and its four levels get finer: keyframes of the whole task, phases between grasps and releases, moments around each sub-event, and 20-frame clips [^1]. The agent accesses them through read-only tools. It reads the coarse levels before planning, then, during execution, goes back into the hierarchy whenever a step needs detail and loads only the clip for its current sub-goal [^1]. One demonstration per task is enough [^1].

Built on RPent, an agentic framework that exposes a frozen VLA as one retryable action, RV-ICL raised task success from 92.6 to 96.5 percent on LIBERO-PRO and from 86.7 to 95.8 percent on LIBERO-Plus [^1].

Why it matters

The method addresses a basic mismatch: what an agent needs from a demonstration changes over a task, from the overall structure while planning to fine detail at each contact [^1]. Letting the agent pull the right level of detail at the right moment keeps context small without losing the information that makes a grasp work, and it does so with no extra training.

Practical example

Say a robot must close a cabinet door in a layout it has not seen before. While planning, its agent reads the overview of a single demonstration: grasp, swing, release. When it reaches the grasp, it loads only the 20-frame clip of that grasp to see exactly how the gripper met the handle, then moves on [^1].

Sources

  1. [1]arXiv — Recursive Video In-Context Learning for Agentic Robot