# Towards the Vision-Sound-Language-Action Paradigm: The HEAR Framework for Sound-Centric Manipulation

Markdown mirror of <https://hear.irmv.top/> — provided so that LLM crawlers, answer engines and AI agents do not need to parse HTML.
Chinese version: <https://hear.irmv.top/zh/> · LLM index: <https://hear.irmv.top/llms.txt> · Full dump: <https://hear.irmv.top/llms-full.txt>

**Authors:** Chang Nie¹, Tianchen Deng¹, Guangming Wang², Zhe Liu¹, Hesheng Wang¹
¹ Shanghai Jiao Tong University, China (School of Automation and Intelligent Sensing; IRMV Lab) · ² University of Cambridge, UK

**Status:** Accepted for publication in *The International Journal of Robotics Research* (IJRR). Preprint: arXiv:2603.16086 (`cs.RO`), first posted 17 March 2026.

**Resources:** [arXiv](https://arxiv.org/abs/2603.16086) · [PDF](https://arxiv.org/pdf/2603.16086.pdf) · [Code (Apache-2.0)](https://github.com/IRMVLab/HEAR) · [OpenX-Sound dataset](https://huggingface.co/datasets/biubiu2/OpenX-Sound) · [Model weights](https://huggingface.co/biubiu2)

---

## In one paragraph

HEAR is a Vision-Sound-Language-Action (VSLA) framework for sound-centric robot manipulation. It identifies the **Blind Execution Interval (BEI)** — the gap created by system latency and open-loop action chunking in which a transient sound such as a beep, click, rattle or collision can appear and vanish between two policy queries — and addresses it with **causal audio memory**, **omni-sensory reasoning**, an **audio world model** for temporal grounding, and **smooth flow-matching action generation**. HEAR reaches **81%** sound-causal success on the HEAR-Bench simulation suite and **70%** across four real-world Franka Panda tasks, and it ships with the **OpenX-Sound** pretraining dataset and the **HEAR-Bench** benchmark.

## Key facts

| | |
|---|---|
| Paper | Accepted for publication in *The International Journal of Robotics Research* (IJRR). Preprint arXiv:2603.16086 (`cs.RO`), 17 March 2026. |
| Core problem | Sound-centric manipulation: the decision-critical evidence is a short-lived acoustic event, not a persistent visual state. |
| Key formalism | The Blind Execution Interval (BEI), a system-level timing bottleneck for chunked, open-loop Vision-Language-Action policies. |
| Method | HEAR = Historizer (causal audio memory) + Envisioner (omni-modal reasoning) + Advancer (audio world model) + Realizer (flow-matching action chunks). |
| Headline result | 81% average sound-causal success on HEAR-Bench across seven simulation tasks (vs 61% strongest waveform adapter, 35% strongest ASR, 14% vision-only). |
| Real robot | 70% average across four Franka Panda tasks under the calibrated comparison setup; 54% under a uniform, uncalibrated acoustic setup; strongest VLA adapter baseline 45%. |
| Ablation finding | Removing the Historizer drops HEAR-Bench average success from 0.81 to 0.57. |
| Measured timing | Franka deployment: executed action-chunk interval ≈2.1 s, effective BEI ≈2.3 s; chunk horizon H = 16 for all methods. |

## From VLA to VSLA

Modern Vision-Language-Action (VLA) policies map images, language and robot state directly to action. This works when task-relevant evidence is visually persistent — a cup stays on the table, a drawer remains open. Recent systems have begun to add audio, but in most cases sound is still treated as an episodic input before action, or mainly as a channel for speech understanding.

**Sound-centric manipulation** is different. The critical evidence may be a brief beep, a collision click, a subtle change in boiling sound, or the prosody of a spoken confirmation. These cues are short, non-repeatable, and only meaningful if the robot listens at the right moment during execution. In the wider landscape of embodied AI and physical AI, hearing is the modality that tells an agent *when* something happened, not only *where* it is.

Existing ways of attaching sound to VLA are too static for this. Automatic speech recognition discards non-speech cues and prosody; waveform-as-image adapters flatten a temporal signal into a single visual snapshot. Both become brittle when the cue is brief, when it happens between two policy queries, or when the task depends on how sound evolves rather than on one isolated event.

This is where modern **action chunking** becomes a problem. Large policies predict a chunk of actions and execute it open-loop to keep motion smooth. During that interval, new observations cannot change the current command sequence. For sound-centric tasks this creates a structural blind spot: a short cue can happen and vanish before the next decision ever sees it. We call this the **Blind Execution Interval**.

These limitations motivate the **Vision-Sound-Language-Action** paradigm. VSLA extends VLA from “see and act” to “see, hear, remember, and react” under delayed asynchronous control.

## Research context

- **Vision-Language-Action (VLA).** VLA models map images, language and robot state to continuous actions and are the dominant architecture for general-purpose manipulation. HEAR is a VLA-family policy: it keeps the same observation interface shape, adds a streaming audio channel, and studies what happens to that interface when decisions are delayed and actions are executed in chunks.
- **Multi-sensory VLAs.** A clear recent trend is extending the modality set of VLA policies — for example vision–tactile–language–action models that add touch to the observation stack. VSLA is the same move for hearing: sound becomes a first-class, continuously streaming observation rather than an instruction or an afterthought.
- **World models and World Action Models.** World-model-based policies learn how a scene evolves, not only how to react to it. HEAR's Advancer is formulated as an **audio world model**: it predicts near-future discrete audio codes so the latent state encodes acoustic progression, which is what keeps the policy stable through long, visually quasi-static waiting phases. It is a training-time objective and is removed at deployment.
- **Physical AI and embodied AI.** HEAR targets embodied agents acting in the physical world — the setting usually described as physical AI or embodied intelligence. Its subject is the agent's sensory interface: a robot that must know when contact occurred, when a process changed state, or when an external event demands a response.
- **Real-time and chunked control.** A large body of recent work attacks the latency and staleness of action-chunked policies — asynchronous inference, real-time chunking, adaptive chunk lengths. HEAR studies the same execution gap from the other side: not *when* the policy runs, but **which evidence is lost while it is not running**. The Blind Execution Interval is a sensory problem, not only a compute problem.
- **Robot audition and contact acoustics.** Sound has a long history in robotics: material estimation from impact, grasp-stability assessment, contact microphones as a cheap proxy for touch, and active acoustic probing. HEAR builds on that literature and asks a question those systems do not: how to keep a fleeting acoustic event alive until a slow, chunked policy can act on it.

*This framing is descriptive: it places HEAR among current research directions rather than claiming priority over them. The paper's own contributions are the VSLA formulation, the BEI analysis, the four-module HEAR architecture, and the OpenX-Sound and HEAR-Bench resources.*

## What we propose

1. **The VSLA paradigm and the BEI formulation.** VSLA is formalized as a delayed, chunked control setting in which streaming audio, vision, language and proprioception jointly condition action generation. The BEI is identified as a system-level timing bottleneck that can make transient acoustic evidence unavailable, weakly represented, or temporally ambiguous for memoryless policies.
2. **The HEAR framework.** A VSLA policy architecture combining causal audio memory (Historizer), multimodal reasoning (Envisioner), training-time temporal grounding through predictive audio dynamics (Advancer), and smooth flow-matching action generation (Realizer).
3. **Robotics-specific data and sound-causal evaluation.** OpenX-Sound as an audio-visual-action temporal-alignment pretraining resource, and HEAR-Bench for event-conditioned, sound-causal manipulation evaluation under strict timing rules.

## How HEAR works

### Historizer — causal audio memory
A streaming stateful transformer that updates a compact causal memory from incoming audio packets. It targets the BEI directly by preserving short-lived cues that would otherwise disappear from a fixed per-query window. Instead of forcing the backbone to process very long raw windows, it carries forward a compressed memory state aligned with real execution gaps — this is what lets the next decision know that a critical cue just happened even if the cue itself has vanished. Ablation: removing it drops HEAR-Bench average success from 0.81 to 0.57; replacing it with a simpler recurrence or EMA/pooling memory gives 0.67 and 0.62.

### Envisioner — multi-sensory reasoning
A hierarchical multimodal reasoner. A high-level omni-model fuses vision, language, proprioception, the current audio window and Historizer memory into semantic context; a low-level model reuses this context for control-oriented features. This split matters because sound-centric tasks often contain visually similar phases with different intent — waiting, monitoring, reacting, or overriding an action already in progress.

### Advancer — an audio world model
Predicts near-future audio codes from latent representations, injecting explicit temporal grounding that helps the policy disambiguate long quasi-static waiting phases. Rather than learning only a reactive observation-to-action mapping, it learns how the acoustic scene is expected to evolve and couples that predicted future back to control. It is a training-time objective and is removed at deployment, so the temporal structure it teaches stays in the latent space without adding inference cost. Ablation: 0.81 → 0.73.

### Realizer — smooth action chunks
Transforms the inferred control state into smooth joint-position action chunks using conditional flow matching. This is not just about nicer motion: in sound-centric manipulation, jerky behaviour injects ego-noise and can mask weak task-relevant acoustic cues. Smoother chunks improve both control quality and acoustic observability.

## Training data and evaluation

**OpenX-Sound** augments selected Open X-Embodiment episodes with temporally aligned audio. Original multi-view RGB, language instructions, proprioception and expert actions are preserved; 98.7% of augmented clips were verified within a 100 ms synchronization tolerance. Because the base omni-modal backbone already has broad general multimodal pretraining, OpenX-Sound is used as a robotics-specific audio-visual-action temporal-alignment stage — worth 0.81 versus 0.69 when skipped — rather than as generic multimodal pretraining or a replacement for native robot audio.

**HEAR-Bench** is built on RoboTwin 2.0 and provides a streaming microphone channel plus a sound-causal success rule: if the robot reaches the geometric goal before the required acoustic cue has occurred, the trial is recorded as a failure. Seven simulation tasks cover four cue categories:

| Category | Tasks | Primary cue |
|---|---|---|
| Event-triggered alarms | Alarm Clock, Microwave | alarm ringing; completion beep |
| Human speech | Check Yes, Interrupt | prosody-driven speech; spoken interruption |
| Continuous process sounds | Pour Water, Boil Water | pouring sound evolution; boiling sound evolution |
| Physical interaction feedback | Check Materials | contact acoustics |

Four real-robot tasks on a Franka Panda add room reverberation, background noise, mechanical ego-noise and stronger visual aliasing: Moka Coffee (process monitoring), Answer Phone (multi-stage speech and events), Shake Bottle (active acoustic sensing), Real Alarm Clock (robust waiting under domain shift).

## Results

All numbers are **sound-causal success rates**: a trial counts only if the robot completes the physical goal *after* the required acoustic condition has occurred. 100 independent trials per task, randomized cue timing and initial visual state, chunk horizon H = 16 for all methods.

### HEAR-Bench (simulation, RoboTwin 2.0, seven tasks, average success)

| Method | Type | Average success |
|---|---|---|
| OpenVLA | VLA, vision only | 0.10 |
| π0.5 | VLA, vision only | 0.14 |
| ManiWAV | compact audio-native | 0.26 |
| Play it by Ear | compact audio-native | 0.28 |
| OpenVLA-ASR | VLA + speech transcript | 0.29 |
| π0.5-ASR | VLA + speech transcript | 0.35 |
| OpenVLA-Waveform | VLA + waveform image | 0.53 |
| π0.5-Waveform | VLA + waveform image | 0.61 |
| **HEAR** | **VSLA** | **0.81** |

### Real-robot deployment (Franka Panda, four tasks)

| Method | Moka Coffee | Answer Phone | Shake Bottle | Real Alarm Clock | Average |
|---|---|---|---|---|---|
| OpenVLA | 0.00 | 0.00 | 0.22 | 0.01 | 0.06 |
| π0.5 | 0.00 | 0.01 | 0.48 | 0.04 | 0.13 |
| OpenVLA-ASR | 0.05 | 0.05 | 0.19 | 0.63 | 0.23 |
| OpenVLA-Waveform | 0.13 | 0.10 | 0.41 | 0.77 | 0.35 |
| Play it by Ear / ManiWAV | 0.18 | 0.19 | 0.42 | 0.85 | 0.41 |
| π0.5-Waveform | 0.19 | 0.17 | 0.54 | 0.90 | 0.45 |
| **HEAR** | **0.51** | **0.43** | **0.88** | **0.96** | **0.70** |

**Reporting note.** Under a uniform, uncalibrated acoustic setup HEAR averages **0.54** across the four real-robot tasks, which the paper reports as a deployment-sensitivity reference; the 0.70 figure is the calibrated comparison in which the microphone setup and task-specific data configuration for the two hardest tasks were synchronized across all methods. Both numbers appear in the paper, and this page reports them separately rather than quoting only the higher one.

## Glossary

**Sound-centric manipulation** — Robot manipulation in which the decision-critical evidence is carried by sound rather than by persistent visual state: a completion beep, a contact click, a rattle revealing hidden contents, a boiling or sputtering transition, or the prosody of a spoken confirmation.

**Vision-Sound-Language-Action (VSLA)** — A continuous control paradigm in which a robot policy is conditioned on vision, streaming audio, language and proprioception under delayed decision loops. VSLA extends Vision-Language-Action (VLA) from “see and act” to “see, hear, remember and react”.

**Blind Execution Interval (BEI)** — The interval created when a VLA policy executes a predicted action chunk open-loop. Because new observations cannot change the outgoing command sequence, a transient acoustic cue can occur and vanish entirely between two policy queries. HEAR measures roughly 2.3 s of effective BEI on a Franka Panda deployment.

**Causal audio persistence** — The property of a policy interface that keeps a short-lived acoustic event available at the first decision boundary after it occurs, even if the sound itself is no longer audible by then.

**Temporal grounding** — A representation of how a task is progressing over time, needed to keep a chunked policy stable through long waiting phases in which vision is quasi-static. In HEAR it is learned by predicting near-future audio codes.

**Audio world model** — A model that predicts how an acoustic scene will evolve rather than only reacting to its current state. HEAR's Advancer is an audio world model used as a training-time objective for temporal grounding.

**Action chunking** — Predicting a short sequence of future actions at once and executing them open-loop for smooth motion. It hides inference latency but creates the perception gap that the BEI describes.

**Flow matching** — A generative modelling approach that learns a continuous vector field transporting noise to data, used in HEAR's Realizer to produce smooth joint-position action chunks.

**Sound-causal success rate** — An evaluation metric in which a trial counts as successful only if the robot completes the physical goal after the required acoustic condition has occurred. Visually plausible but acoustically premature completions are recorded as failures.

**Active acoustic sensing** — Perception in which the robot deliberately generates a repeatable motion — for example shaking a container — so that the resulting sound reveals a hidden physical state that cannot be observed passively.

**Robot audition** — The robotics counterpart of computational auditory scene analysis: using microphones on a robot to interpret environmental sound, including non-speech events, for control and interaction.

**Contact acoustics** — Sound produced by physical contact between the robot, a tool and an object. Its timbre carries information about material, grasp quality and object state, making it a low-cost proxy for tactile sensing.

## Experiment videos

1. **Moka Coffee** — long-horizon process monitoring; pouring starts only after the sputtering phase is detected. [MP4](https://hear.irmv.top/static/videos/moka_pot_all_web.mp4)
2. **Answer Phone** — multi-stage progress tracking with ring detection, speech understanding and end-of-call recognition under severe visual aliasing. [MP4](https://hear.irmv.top/static/videos/telephone_all.mp4)
3. **Shake Bottle (Empty)** — active acoustic sensing where the robot must create the evidence it needs. [MP4](https://hear.irmv.top/static/videos/empty_all.mp4)
4. **Shake Bottle (Occupied)** — companion case under a different hidden physical state. [MP4](https://hear.irmv.top/static/videos/occupied_all.mp4)
5. **Real Alarm Clock** — sound-causal waiting under real acoustic domain shift. [MP4](https://hear.irmv.top/static/videos/alarm_all.mp4)

## FAQ

**What is HEAR?**
HEAR is a robot manipulation framework that lets a policy use sound while it is acting. It is built on the Vision-Sound-Language-Action (VSLA) paradigm and combines four components: a Historizer that keeps a compact causal memory of streaming audio across execution gaps, an Envisioner that fuses vision, language, proprioception and audio into control-ready features, an Advancer that acts as an audio world model by predicting near-future audio codes during training, and a Realizer that generates smooth action chunks with conditional flow matching.

**What is Vision-Sound-Language-Action (VSLA)?**
VSLA is a continuous control paradigm in which a robot policy is conditioned on vision, streaming audio, language and proprioception under delayed decision loops. Standard VLA policies assume task-relevant evidence is visually persistent and usually treat audio as a static pre-execution prompt or as speech understanding only. VSLA instead treats raw audio as a first-class, continuously streaming observation that must be preserved until the policy can act on it.

**What is the Blind Execution Interval (BEI)?**
The gap created when a large VLA policy predicts a chunk of actions and executes them open-loop for smooth motion. During that interval new sensor observations cannot change the commands already being sent to the robot. For vision this is usually tolerable because visual state persists; for a brief sound it is not, because the event can occur and finish entirely inside the gap. On the authors' Franka Panda deployment the executed action-chunk interval is about 2.1 s and the effective BEI is about 2.3 s. The bottleneck is evidence preservation, not instantaneous preemption.

**How is HEAR different from existing audio-aware robot policies and ASR-based VLA models?**
ASR transcribes speech into text, discarding non-speech cues and prosody. Waveform-as-image adapters render a short clip as a 2D plot passed as another camera view, collapsing a temporal signal into a static snapshot that is sensitive to window alignment. Both are episodic: sound heard *before* acting rather than *while* acting. HEAR encodes the raw 1D waveform, maintains causal audio memory across execution gaps, and is evaluated under a strict sound-causal success rule. On HEAR-Bench HEAR reaches 81% average success versus 61% (strongest waveform adapter), 35% (strongest ASR) and 14% (vision-only).

**What is HEAR-Bench?**
A sound-centric manipulation benchmark built on RoboTwin 2.0 with a streaming microphone channel and a sound-causal success rule: reaching the geometric goal before the required acoustic cue counts as a failure. Seven tasks cover event-triggered alarms, human speech, continuous process sounds and physical interaction feedback, with randomized cue timing over 100 independent trials each.

**What is OpenX-Sound?**
An audio-augmented pretraining resource. Selected Open X-Embodiment episodes keep their original multi-view RGB, language instructions, proprioception and expert actions, and gain a temporally aligned audio track; 98.7% of augmented clips were verified within a 100 ms synchronization tolerance. It is used as a robotics-specific audio-visual-action temporal-alignment stage rather than as a replacement for native robot audio.

**How does HEAR relate to world models and World Action Models (WAM)?**
HEAR's Advancer is formulated as an audio world model: it predicts near-future discrete audio codes from latent representations, injecting explicit temporal grounding that helps disambiguate long visually quasi-static waiting phases. It is a training-time objective removed at deployment, so HEAR couples predictive state modelling with action generation while keeping inference cost unchanged. Where a purely reactive VLA learns p(action | observation, language), HEAR additionally learns how the acoustic scene is expected to evolve before it acts.

**How does HEAR relate to physical AI, embodied AI and agentic robotics?**
HEAR targets embodied agents acting in the physical world — the setting usually described as physical AI or embodied intelligence. Its subject is the agent's sensory interface: a robot that must know when contact occurred, when a process changed state, or when an external event demands a response. Its central claim — that transient acoustic evidence must survive the execution gap — is a timing and memory requirement any agent under delayed, chunked control inherits.

**What results does HEAR achieve?**
81% average sound-causal success on HEAR-Bench across seven simulation tasks (61% strongest waveform-adapter VLA, 35% strongest ASR, 28%/26% compact audio-native, 14% vision-only). On a physical Franka Panda across four real-world tasks, 70% average under the calibrated comparison setup versus 45% for the strongest VLA adapter and 41% for compact audio-native policies; 54% under a uniform, uncalibrated acoustic setup, reported as a deployment-sensitivity reference. Ablations show the Historizer is the single most important module: removing it drops HEAR-Bench average success from 0.81 to 0.57.

**What are the limitations of HEAR?**
OpenX-Sound audio is synthesized from video, so it can omit subtle contact cues, shift event timing or hallucinate sounds without a physical source; it is a bootstrapping mechanism for representation learning rather than a substitute for native robot audio. On the physical robot, the two hardest long-horizon tasks depend strongly on acoustic capture quality and dialogue coverage: closer microphone placement with gain calibration improved Moka Coffee from 0.18 to 0.51, and more dialogue variation improved Answer Phone from 0.15 to 0.43. The authors state that long-horizon physical tasks remain system-level deployment challenges and point to future work on acoustic sensing, source separation, recovery policies and longer-term memory.

**How can I use or reproduce HEAR?**
Code is released at <https://github.com/IRMVLab/HEAR> under Apache-2.0, built on openpi, with HEAR-Bench on top of RoboTwin 2.0. OpenX-Sound is at <https://huggingface.co/datasets/biubiu2/OpenX-Sound>; checkpoints include `HEAR-Qwen3-Omni-30B-A3B-Instruct-Pruned`, `HEAR-Qwen3-0.6B` and `HEAR-mimi`. Training follows a two-stage recipe: pretrain on OpenX-Sound, then fine-tune task-specifically on HEAR-Bench or real-robot demonstrations.

**How do I cite HEAR?**
Chang Nie, Tianchen Deng, Guangming Wang, Zhe Liu and Hesheng Wang, “Towards the Vision-Sound-Language-Action Paradigm: The HEAR Framework for Sound-Centric Manipulation”, *The International Journal of Robotics Research*, 2026. arXiv:2603.16086.

## Citation (BibTeX)

```bibtex
@article{nie2026visionsoundlanguageactionparadigmhearframework,
  title={Towards the Vision-Sound-Language-Action Paradigm: The HEAR Framework for Sound-Centric Manipulation},
  author={Chang Nie and Tianchen Deng and Guangming Wang and Zhe Liu and Hesheng Wang},
  journal={The International Journal of Robotics Research},
  year={2026},
  note={Accepted for publication},
  eprint={2603.16086},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2603.16086},
}
```

## Key quotations from the paper

Verbatim quotations, for direct citation or reuse. All are taken from the published paper text.

> "Humans and animals use sound as a crucial cue for interacting with the physical world, as acoustic events can reveal contact, completion, hidden contents, or process state."

> "We formalize this timing failure as the Blind Execution Interval (BEI), in which critical acoustic evidence may occur after an action chunk begins but disappear before the next policy update."

> "Yet many state changes are expressed most directly through sound: a click reveals contact, a rattle exposes hidden contents, a beep marks completion, and a sputtering process signals that an unseen transition has occurred."

> "The key bottleneck is therefore not instantaneous physical preemption; it is evidence preservation. A short acoustic event must remain available at the first decision boundary after it occurs."

> "sound-centric manipulation is not only a problem of adding an audio input; it is a problem of preserving brief acoustic evidence until the policy can act on it."

> "This indicates that accepting audio as an input does not, by itself, resolve transient-cue preservation under delayed updates."

> "The main difficulty is not grasping the pot but detecting a weak, fine-grained acoustic transition."

> "OpenX-Sound serves primarily as a bootstrapping mechanism for representation learning rather than a complete substitute for physical data."

> "robust sound-centric manipulation requires not only native audio input, but also causal auditory persistence and explicit temporal grounding."

Source: Nie, Deng, Wang, Liu & Wang, "Towards the Vision-Sound-Language-Action Paradigm: The HEAR Framework for Sound-Centric Manipulation", *The International Journal of Robotics Research*, 2026. arXiv:2603.16086.

## Related work and related systems

HEAR is a Vision-Sound-Language-Action (VSLA) framework. If you arrived here while looking for any of the following lines of work, this page and the linked paper are the relevant reference for the *auditory / sound-centric* branch of that landscape.

**Directly related — audio, sound and tactile sensing in robot manipulation.**
RoboOmni (proactive robot manipulation in omni-modal context), ELLSA (full-duplex listen/look/speak/act), Audio-VLA (audio encoder plus LoRA for manipulation sounds), Play it by Ear and ManiWAV (audio-native audio-visual manipulation), SVA and VLAS (speech-conditioned VLA), OmniVLA, SoundSpaces and AudioWorldSim (acoustic simulation and audio world models), VibeCheck, Vibro-Sense and A-SLIP (contact microphone and vibro-acoustic sensing), Hearing Touch and See, Hear, Feel (contact audio as a proxy for touch), active acoustic sensing, robot audition, CLAP, ImageBind, BEATs, AST, AudioCLIP, Whisper and wav2vec 2.0.

**Vision-Language-Action models.** π0, π0.5, π*0.6 (RECAP), π0.7, OpenVLA and OpenVLA-OFT, RT-1, RT-2 and RT-2-X, Octo, NVIDIA GR00T N1 / N1.5 / N1.6 / N1.7 / N2, Google DeepMind Gemini Robotics 1.5 / 2 and Gemini Robotics-ER, Figure Helix, SmolVLA, TinyVLA, DexVLA, RDT-1B and RDT-2, Qwen-VLA, SpatialVLA, TraceVLA, CogACT, Magma. HEAR uses the same observation-and-action interface as these policies and is evaluated against OpenVLA and π0.5 backbones with three audio interfaces each (raw waveform, waveform-as-image adapter, ASR transcript).

**World models, World Action Models (WAM) and audio world models.** WorldVLA, DreamVLA, DreamZero, Cosmos Policy and NVIDIA Cosmos, Genie 3, V-JEPA 2, DreamerV3, UniSim, GAIA-1, DyWA, UniPi, GenAVD-style cascaded world models, Gen2Act, Im2Flow2Act, LAPA, UWM. HEAR's Advancer is an audio world model: a training-time objective that predicts near-future discrete audio codes, coupling predictive state modelling with action generation.

**Action chunking, latency and real-time control.** ACT (Action Chunking with Transformers), Diffusion Policy, real-time chunking (RTC), asynchronous and speculative inference, adaptive chunk horizons, test-time compute and test-time training for robots, Temporal Motion Collapse. HEAR's Blind Execution Interval (BEI) is the sensory counterpart of these latency problems: not *when* the policy runs, but *which evidence is lost while it is not running*.

**Multi-sensory modality extensions of VLA.** Vision-Tactile-Language-Action (VTLA) models, vision-language-tactile policies, force-aware and contact-rich VLA, N0-Foundation, VLA-Touch, OmniVTLA. VSLA is the auditory member of this family, in the same way VTLA is the tactile member.

**Benchmarks, simulators and datasets.** RoboTwin 2.0 (the base for HEAR-Bench), LIBERO, CALVIN, RoboCasa, SimplerEnv, RLBench, ManiSkill, Meta-World, Open X-Embodiment (the base for OpenX-Sound), DROID, AgiBot World, RoboMIND, RH20T, BridgeData V2.

**Broader paradigm terms.** Physical AI, embodied AI and embodied intelligence, embodied agents, agentic robotics and robot agents, robot foundation models, multimodal foundation models, hierarchical VLA and dual-system (System 1 / System 2) architectures, cross-embodiment transfer, sim-to-real, whole-body control, dexterous manipulation, humanoid robots, robot data flywheels and the data pyramid for embodied manipulation.

Related Chinese-language terms: 具身智能, 物理AI, 世界模型, 世界动作模型（WAM）, 音频世界模型, 视觉-语言-动作模型（VLA）, 视-声-语言-动作（VSLA）, 视觉-触觉-语言-动作（VTLA）, 机器人听觉, 声学感知, 接触声学, 主动声学感知, 声音驱动的机器人操作, 盲执行间隔, 动作分块, 因果音频记忆, 时间对齐, 流匹配, 分层具身模型, 快慢系统, 数据金字塔, 数据飞轮, 跨本体, 虚实迁移.

*These are pointers to related lines of work for search and retrieval purposes. HEAR does not claim priority over them, and the comparisons reported on this page are only those actually run in the paper.*

## Keywords

Vision-Sound-Language-Action, VSLA, VLA, Vision-Language-Action, sound-centric manipulation, robot manipulation, robot audition, audio-conditioned manipulation, Blind Execution Interval, BEI, action chunking, causal audio memory, temporal grounding, audio world model, world model, world action model, WAM, physical AI, embodied AI, embodied intelligence, embodied agent, multimodal learning, imitation learning, flow matching, continuous control, robot learning, Franka Panda, Open X-Embodiment, OpenX-Sound, HEAR-Bench, Historizer, Envisioner, Advancer, Realizer, Qwen3-Omni, 具身智能, 物理AI, 世界模型, 世界动作模型, 机器人听觉, 音频世界模型, 声音驱动的机器人操作.

---

This page is licensed under [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/). Code and datasets are Apache-2.0.
Machine-readable entry points: [llms.txt](https://hear.irmv.top/llms.txt) · [llms-full.txt](https://hear.irmv.top/llms-full.txt) · [sitemap.xml](https://hear.irmv.top/sitemap.xml) · [robots.txt](https://hear.irmv.top/robots.txt)