Decoupling Planning and Control for Instructable Agents

Recent work on vision-language(-action) agents shows that VLMs are strong at high-level reasoning but struggle to realize plans as reliable low-latency action sequences, while world-model controllers excel at fast observation-to-action control but lack open-ended task guidance. In this work, we combine these strengths by conditioning a learned world-model controller on language so that it can act autonomously at high frequency conditioned on sparse, higher-latency textual instructions generated by vision-language models (VLMs). Our system, Instruct-to-Act, includes an instructable controller that autoregressively generates high-frequency actions and can either follow language instructions from an instruction agent, or self-operate in a high-throughput environment.

To train controllers to be language-instructable, we relabel segments of controller policy rollouts with instructions and optimize a behavior-cloning objective. Our framework easily supports extension to multi-agent settings that enable agent communication between VLMs using trained controllers as actuators without relying on Multi-Agent Reinforcement Learning algorithms. We report results on various embodied environments and tasks, scaling trends with larger controllers and VLMs, and ablations on instruction cadence, planning frequency, and online vs. offline planning latency. The results show that with our decoupled architecture, Instruct-to-Act can flexibly switch to different VLMs and scale well to multi-agents and longer chains of reasoning achieving state-of-the-art performance on six tasks.

Method Overview

Instruct-to-Act system architecture diagram

Key Approach

Decoupled Architecture: Separate high-level planning (VLM) from low-level control (trained world model), enabling real-time execution with sophisticated reasoning
Language-Conditioned Controller: Train environment-specific controllers based on Dreamer (world-model RL) that can follow natural language instructions
Post-hoc Instruction Supervision: Automatically annotate replay buffer segments with language descriptions using VLMs, enabling behavior cloning without manual labeling
Asynchronous Online Planning: VLM planners continuously reason in the background while controllers execute actions, dramatically reducing latency
Multi-Agent Extension: Scale to multiple agents with parameter-shared controllers and language-based coordination, no specialized MARL algorithms needed

Citation

If you find this work useful, please consider citing:

@inproceedings{tang2026instructtoact,
  title     = {Decoupling Planning and Control for Instructable Agents},
  author    = {Tang, Zineng and Allen, Kelsey R. and van Steenkiste, Sjoerd
               and Dasgupta, Ishita and Suhr, Alane},
  booktitle = {Conference on Language Modeling (COLM)},
  year      = {2026}
}