Soundify: Matching Sound Effects to Video
Soundify matches sound effects to video, synchronizes them with the action, and adjusts panning and volume to create spatial audio.
Summary
In the art of video editing, sound helps add character to an object and immerse the viewer within a space. Through formative interviews with professional editors (N=10), we found that the task of adding sounds to video can be challenging. This paper presents Soundify, a system that assists editors in matching sounds to video. Given a video, Soundify identifies matching sounds, synchronizes the sounds to the video, and dynamically adjusts panning and volume to create spatial audio. In a human evaluation study (N=889), we show that Soundify is capable of matching sounds to video out-of-the-box for a diverse range of audio categories. In a within-subjects expert study (N=12), we demonstrate the usefulness of Soundify in helping video editors match sounds to video with lighter workload, reduced task completion time, and improved usability.
Video
Example Results
Cars
Cooking + Fire Crackle
Ducks + Water Ripple
Footsteps + Tram
Helicopter
Lightning
Seagulls + Crows
Washing Machine
Waterfall
Waves
Citation
@inproceedings{lin2023soundify,
author = {Lin, David Chuan-En and Germanidis, Anastasis and Valenzuela, Crist{\'o}bal and Shi, Yining and Martelaro, Nikolas},
title = {Soundify: Matching Sound Effects to Video},
booktitle = {Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology},
series = {UIST '23},
year = {2023},
articleno = {18},
numpages = {13},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
doi = {10.1145/3586183.3606823},
url = {https://doi.org/10.1145/3586183.3606823},
location = {San Francisco, CA, USA}
}