Multi-Sensory Robot Policies
Multi-sensory robot policies extend a vision-language-action model with modalities beyond vision — typically touch, force or sound. The motivation is straightforward: cameras cannot observe contact, force or hidden internal state, so tasks that depend on them need another channel. The engineering lesson is less obvious: each modality has its own temporal structure, and an interface that suits vision does not automatically suit the others.
The families
Vision-tactile-language-action (VTLA). Adds tactile sensing, usually from fingertip sensors or a tactile skin, to the observation stack. Tactile data is high-dimensional, local and fast-changing, which makes it informative for contact-rich manipulation and awkward to feed to a model built for camera frames.
Force and torque. Wrist or joint force sensing gives a lower-dimensional but very direct measure of contact. Often added as an additional input vector rather than an image-like modality.
Audio — VSLA. Adds streaming sound. Unlike touch and force, audio carries information about events that are not happening at the robot's own end-effector: an appliance finishing, a phone ringing, a liquid reaching a boil.
Why the temporal structure matters more than the modality
The standard VLA assumption is that observations can be sampled at low frequency because the world changes slowly. That assumption is reasonable for cameras and disastrous for modalities whose information arrives in short bursts.
Tactile and force signals are dense but strongly correlated with the robot's own motion, so a policy generally knows when to expect them. Audio is different: task-relevant acoustic events are sparse, asynchronous, and often generated by something other than the robot. They can occur at any moment, including during an open-loop execution chunk, and they do not repeat.
The design consequence
Adding a modality is not the same as adding an input channel. A modality whose evidence is transient requires the interface to have memory: something that accumulates what arrived and keeps it available until a decision can use it. Without that, the additional modality is present in the API and absent in effect.
Where HEAR fits
HEAR is the auditory member of this family. It pairs a streaming audio interface with causal memory (the Historizer), a reasoning module that fuses it with vision, language and proprioception (the Envisioner), and an evaluation protocol that rejects actions completed before the required sound occurred. It is designed to be combined with, not to replace, tactile and force sensing.