PronounSE: SFX Synthesizer from Language-Independent Vocal Mimicry

Riki Takizawa^{1, 4}, Shigeyuki Hirai², Asako Kanezaki³, Hitoshi Suda^4
1Graduate School of Kyoto Sangyo University, Japan
²Kyoto Sangyo University, Japan
³Institute of Science Tokyo, Japan
⁴National Institute of Advanved Industrial Science and Technology, Japan

Abstract

This study proposes PronounSE, a sound effect synthesis method that utilizes only non-verbal vocal expressions mimicking sound effects, without requiring prior knowledge or experience in sound production techniques. Focusing on human articulatory capabilities, PronounSE learns the correspondence between variations in vocal mimicry and actual sound effects, enabling the generation of sound effects that reflect the nuanced characteristics of vocal input. In this study, we constructed a new dataset centered on explosion sounds and trained PronounSE accordingly, successfully synthesizing explosion sounds that reflect the nuances of vocal mimicry. Furthermore, we designed novel evaluation metrics and conducted comparative experiments with existing methods.

Synthesis Sample

Input Vocal Mimicry: windy nuance with the explosion

Synthesized Audio: nuance-reflective synthesized sound

Comparative Evaluation with Other methods

In this study, we introduced new subjective evaluation metrics focusing on the similarity between the reference and synthesized sounds, as well as the extent to which vocal mimicry nuances are reflected in the synthesized outputs. In addition, we conducted conventional subjective evaluations on the naturalness and sound quality of the synthesized sounds. For comparison, we evaluated two Audio-to-Audio methods: T-Foley [1] and Stable Audio 2.0 [2].

Using an evaluation dataset consisting of vocal mimicry samples and reference sounds from six speakers, each method was used to synthesize audio. An example of the reference sound, the six vocal imitations, and the corresponding synthesized sounds is shown below.

Reference

Vocal Mimicry and Synthesized Sounds

Objective Fidelity Evaluation with FAD [3] and Cosine Similarity
  Objective evaluation results on FAD between the reference and synthesized sound sets, and the average cosine similarity between the embedding vectors of corresponding sound pairs.

Subjective Fidelity Evaluation (Fidelity MOS ± 95% CI)
  5-point MOS resuts on the fidelity of synthesized sounds to reference sounds.

Subjective Nuance Reflection Evaluation (Attack MOS ± 95% CI, Release MOS ± 95% CI, Overall MOS ± 95% CI)
  5-point MOS results on the nuance reflection of vocal mimicry in the synthesized sounds, evaluated from three perspectives: the attack phase, the release phase, and the overall impression.

Subjective Naturalness and Sound Quality Evaluation (Naturallness MOS ± 95% CI, Quality MOS ± 95% CI)
  5-point MOS results on the naturalness and audio quality of the synthesized sounds.

Links

Citation

If you use this work, please cite our paper:

   @article{hoge,
        title   = {hogehoge},
        author  = {Riki Takizawa, Shigeyuki Hirai, Asako Kanezaki, Hitoshi Suda},
        journal = {hoge},
        volume  = {hoge},
        number  = {hoge},
        pages   = {hoge},
        year    = {hoge},
        doi     = {hoge}
    }

Copied!

Reference

[1]: Y. Chung, J.Lee, and J. Nam. "T-FOLEY: A Controllable Waveform-Domain Diffusion Model for Temporal-Event-Guided Foley Sound Synthesis". In: ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE.
[2]: Stability AI. "Stable Audio 2.0". https://stability.ai/stable-audio.
[3]: ilgour, K., Zuluaga, M., Roblek, D. and Sharifi, M.: Fr´echet Audio Distance: A Reference-Free Metric for Evaluating Music Enhancement Algorithms, Proceedings of the 20th Annual Conference of the International Speech Communication Association (INTERSPEECH), pp. 2350–2354