OpenX-Sound: Audio-Augmented Robot Trajectories

Every episode preserves the original multi-view RGB, the language instruction, the proprioceptive state and the expert action sequence, with an added audio track aligned to the video. 98.7% of augmented clips were verified within a 100 ms synchronisation tolerance. Episodes with semantically inconsistent audio, implausible event timing or obvious synchronisation errors were corrected or removed during review.

The authors are explicit about the scope. Because the base omni-modal backbone already has broad multimodal pretraining on general audio-video data, OpenX-Sound is used as a robotics-specific audio-visual-action temporal-alignment stage — not as generic multimodal pretraining, and not as a replacement for native robot audio. Skipping this stage entirely costs 0.81 to 0.69 on HEAR-Bench, which the paper reads as evidence that the gain is about robotic audio-visual-action coupling rather than scale.

The acknowledged limitation is the synthetic-to-real gap: video-to-audio generation can omit subtle contact cues, shift event timing, or produce sounds without a physical source. For a task as timing-sensitive as sound-causal manipulation, that caps what purely offline pretraining can achieve, which is why fine-tuning on real recordings remains necessary.

The dataset is released on Hugging Face. Base trajectories derive from Open X-Embodiment; consult that project's licensing for the underlying episodes.