Soundify: Matching Sound Effects to Video

David Chuan-En Lin¹Anastasis Germanidis²Cristóbal Valenzuela²Yining Shi²Nikolas Martelaro¹

¹Carnegie Mellon University

²Runway

UIST Paper NeurIPS Paper Video Talk Demo Code

Soundify system pipeline — Soundify assists users in matching sound effects (in bold) and ambients (in italics) to video, and helps dynamically adjust panning and volume by localizing "sound emitters."

Summary

In the art of video editing, sound helps add character to an object and immerse the viewer within a space. Through formative interviews with professional editors (N=10), we found that the task of adding sounds to video can be challenging. This paper presents Soundify, a system that assists editors in matching sounds to video. Given a video, Soundify identifies matching sounds, synchronizes the sounds to the video, and dynamically adjusts panning and volume to create spatial audio. In a human evaluation study (N=889), we show that Soundify is capable of matching sounds to video out-of-the-box for a diverse range of audio categories. In a within-subjects expert study (N=12), we demonstrate the usefulness of Soundify in helping video editors match sounds to video with lighter workload, reduced task completion time, and improved usability.

Video

About 1 min read

Example Results

Cars

Cooking Fire Crackle

Ducks Water Ripple

Footsteps Tram

Helicopter

Lightning

Seagulls Crows

Washing Machine

Waterfall

Waves

Citation

@inproceedings{lin2023soundify,
  title     = {Soundify: Matching sound effects to video},
  author    = {Lin, David Chuan-En and Germanidis, Anastasis and Valenzuela, Crist{\'o}bal and Shi, Yining and Martelaro, Nikolas},
  booktitle = {Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology},
  pages     = {1--13},
  year      = {2023}
}