# Vision-Sound-Language-Action (VSLA)

> Vision-Sound-Language-Action (VSLA) is a continuous control paradigm in which a robot policy is conditioned on vision, streaming audio, language and proprioception under delayed decision loops. It extends Vision-Language-Action (VLA) from “see and act” to “see, hear, remember and react” by treating raw audio as a first-class, continuously streaming observation rather than a pre-execution prompt.

Source: https://hear.irmv.top/concepts/vision-sound-language-action/ · Language: en

## Why the “S” needs its own paradigm

Adding audio to a VLA policy is not symmetric with adding another camera. Visual state persists: a cup stays on the table across many control cycles, so a slightly stale image is still useful. Acoustic evidence is the opposite — it is concentrated into short, non-repeatable events. A click, a rattle, a completion beep or a boiling transition may last a few hundred milliseconds and never recur. If the policy samples that modality the way it samples images, the evidence is simply gone.

VSLA names the setting in which that asymmetry is taken seriously: streaming audio is part of the observation interface, and the interface must preserve what it hears until a decision can use it.

## The timing problem VSLA formalizes

Large robot policies cannot run at control frequency, so they predict an action chunk and execute it open-loop. During that interval new observations cannot change the outgoing commands. The paper calls this the Blind Execution Interval (BEI). For vision it is tolerable; for a short sound it can be fatal. Re-querying faster shortens the gap but cannot remove it, because sensing, preprocessing, inference, communication and actuation latency all remain. The bottleneck is therefore evidence preservation, not instantaneous preemption.

## Sibling paradigms

VSLA belongs to the same family as vision-tactile-language-action (VTLA) models, which add touch to the observation stack. Both extend the modality set of a VLA policy without abandoning its structure. A robot could in principle carry both.

## Where HEAR fits

HEAR is the instantiation of VSLA presented in the IJRR paper. Its four components map onto the paradigm's requirements: the Historizer provides causal persistence across the BEI, the Envisioner performs multi-sensory reasoning, the Advancer supplies temporal grounding as an audio world model, and the Realizer generates smooth action chunks.

## Frequently asked questions

**What does VSLA stand for?**

Vision-Sound-Language-Action. It is the auditory sibling of Vision-Language-Action (VLA) and Vision-Tactile-Language-Action (VTLA).

**Is VSLA just VLA with audio?**

No. The difference is temporal, not modal. VLA policies assume task-relevant evidence is visually persistent and can be sampled at low frequency. VSLA treats streaming audio as an observation that must survive delayed, chunked execution — which requires memory and timing mechanisms that a plain extra input channel does not provide.

**Who introduced VSLA?**

The paradigm is formalized in the HEAR paper by Chang Nie, Tianchen Deng, Guangming Wang, Zhe Liu and Hesheng Wang, accepted for publication in The International Journal of Robotics Research (2026), arXiv:2603.16086.

---

Chang Nie, Tianchen Deng, Guangming Wang, Zhe Liu and Hesheng Wang, “Towards the Vision-Sound-Language-Action Paradigm: The HEAR Framework for Sound-Centric Manipulation”, The International Journal of Robotics Research, 2026. arXiv:2603.16086.