Robot Manipulation Research Landscape

The dominant architecture for general-purpose manipulation. A large pretrained multimodal backbone is conditioned on camera images, a language instruction and proprioception, and predicts actions directly. The field's shared engineering problem is that these models are expensive, so they cannot run at control frequency — which leads directly to the next two sections.

To hide inference latency and keep motion smooth, policies predict a short sequence of future actions (“a chunk”) and execute it open-loop. Diffusion policy and flow matching are the two dominant generative recipes for producing those chunks, with flow matching now the more common default because it needs fewer integration steps. Chunking solves the smoothness problem and creates a perception problem: during the chunk, new sensor readings cannot change the commands already being sent.

World-model-based approaches learn the dynamics of a scene, often by predicting future observations in pixel space, latent space or token space. When the same model also emits actions, the resulting system is increasingly called a World Action Model (WAM): it targets the joint distribution of future state and action rather than the reactive mapping from current observation to action. The practical appeal is generalisation; the practical difficulty is that predicted futures must be physically consistent, and errors compound over long horizons.

Vision alone cannot observe contact, force or hidden internal state. A growing family of policies adds modalities: vision-tactile-language-action (VTLA) for touch, force-aware policies for contact-rich tasks, and — the branch HEAR works in — audio. Sound is unusual among these modalities because it is temporally sparse: it carries little information during quiet periods and concentrates it into very short events.

Work on asynchronous inference, real-time chunking, adaptive chunk horizons and speculative execution all targets the same issue: a chunked policy's decisions are stale by construction. These methods change when the policy runs. HEAR addresses the complementary question of what sensory evidence is destroyed in the meantime, and shows that for transient acoustic events the answer is “all of it” unless the interface is changed.

Progress in this field tracks the availability of large cross-embodiment datasets and of benchmarks that measure the right thing. Standard manipulation benchmarks evaluate final geometric success, which cannot distinguish a policy that listened from one that happened to move correctly. Sound-centric manipulation needs timing-aware success rules: HEAR-Bench marks a trial as failed if the robot reaches the goal before the required acoustic cue has occurred.

Sound-centric manipulation is a narrow but well-defined slice: tasks whose correctness depends on when a sound occurred. It matters because audition is the modality that reports events rather than states — contact, completion, transitions, hidden contents — and because those events are exactly what a chunked, delayed policy tends to miss.