Feedback háptico para acessibilidade de vídeo em dispositivos móveis

Abstract

Deafness compromises the perceptual experience of individuals during audiovisual content consumption, depriving them of sound elements that carry emotion and meaning within video scenes. This work proposes a system, composed of a web frontend, an audio processing backend, and an Android application, that aims to improve this experience through vibrations synchronized with the video audio. The user loads and plays a video on the frontend, while the backend extracts the audio signal and converts it into a sequence of vibration patterns, based on energy envelopes, onset detection, and shaping curves tuned to the content type. For musical content, the system applies AI-based source separation (the Demucs model) to isolate the vocals before conversion, making the vibration follow the most recognizable element of the song. The patterns are transmitted in real time to the Android application, which triggers the device’s vibration motor in synchrony with playback. The system was evaluated through five objective experiments, which showed low synchronization latency (around 30 ms), faithful reproduction of the audio’s rhythm and intensity, and a measurable benefit of AI separation on music vibration. Moreover, this solution requires no additional hardware, leveraging the vibration motor already present in smartphones.