Baton: A Coding Agent Learns the Handoffs Between Robot Skills
Robot skills that work alone often fail when chained. Baton has a coding agent explore each subtask and its handoffs, lifting task success on a household benchmark from 46.1 to 63.5 percent.
TL;DR
- Baton, from USC and UCF, lets an LLM coding agent steer a frozen robot policy by exploring and remembering each subtask and its handoffs [^1].
- On the RoboMemArena benchmark, Baton completed 63.5 percent of tasks, against 46.1 percent for the best earlier methods [^1].
Background
Vision-language-action (VLA) models map camera images and instructions directly to robot actions. OpenVLA, for instance, is a 7-billion-parameter open model trained on 970,000 real-world robot demonstrations [^2]. Such models increasingly master single skills, but long tasks that chain those skills still fail. The policy π0.5 succeeds 97 percent of the time on LIBERO's single-task suites, yet on thousand-step tasks it clears 38.7 percent of the intermediate stages and finishes only 21.5 percent of the tasks [^1]. Errors compound [^1].
What happened
One workaround puts a coding agent in charge: the agent plans in language, moves the arm through free space with analytic motion primitives and calls the frozen VLA only for contact-rich steps [^1]. On long tasks, the authors say, that recipe has two problems. Exploring whole tasks gets exponentially more expensive with each added stage, and a failed attempt doesn't reveal which stage caused it. And the agent has no notion of transitions, so a subtask can succeed in a way its successor can't use [^1].
Baton, from Bingxin Xu, Yuzhang Shang and Emilio Ferrara, makes the subtask the unit of exploration [^1]. Each subtask is explored on its own and its solution stored in memory; long tasks are then assembled from those solutions. Exploration cost grows linearly with the number of stages, and every failure is tied to one stage [^1]. A transition-aware memory covers the boundaries. A verifier agent calls the VLA only after the wrist camera confirms the scene is ready. A handoff step restores the starting state a subtask expects when its predecessor disturbed it. And a lookahead step picks an execution strategy whose outcome the next subtask can build on, such as a grasp that keeps a drawer handle reachable for closing it again [^1].
RoboMemArena consists of 26 household tasks in kitchen scenes, each chaining 3 to 9 checked stages over more than 1,000 control steps on average [^1]. For a controlled comparison, Baton and the coding-agent baseline Harness VLA used identical parts: the same frozen VLA checkpoint without fine-tuning, SAM-3 for segmentation, GPT-5.6 Sol as the agent and 20 exploration episodes per task [^1]. Baton finished 63.5 percent of tasks and 82.9 percent of stages, against 23.1 and 37.9 percent for Harness VLA. The best earlier methods, which were trained on the benchmark's data, finished 46.1 percent of tasks [^1]. On a PiPER-X robot arm, Baton moved two cans onto a plate in turn, or a can and a bottle, in 47.5 percent of 40 trials, against 27.5 percent for Harness VLA and 7.5 percent for π0.5 [^1].
Why it matters
The biggest gain came where transitions matter most. On occlusion tasks, which mix operations such as opening and closing drawers and microwaves, Harness VLA completed none: within the 20-episode budget, exploring whole tasks rarely reached a first success, so nothing was written to memory. Baton completed half of them [^1]. On sequence tasks, where subtasks are only loosely coupled, the gain was modest [^1]. Because what Baton learns is written in language and attached to transitions rather than to tasks, a handoff learned once can be reused by every task that crosses the same boundary, without retraining the robot policy [^1].
Practical example
Say a kitchen robot has to open a drawer, take out a can, put it in the microwave and close the drawer again. Each step works on its own, but the grasp used to open the drawer leaves the handle out of reach for closing it. A system like Baton tries each step separately, sees that this boundary fails, and stores a lookahead rule to grasp the handle so it stays reachable for the closing motion [^1]. The next task that crosses the same boundary reuses that rule instead of running into the problem again [^1].
Related gear
We recommend this book because it provides the foundational mathematical and algorithmic framework for physical state estimation and robot manipulation.