Emotion Recognition in Speech using Cross-Modal Transfer in the Wild

Citations Over TimeTop 1% of 2018 papers

Abstract

Obtaining large, human labelled speech datasets to train models for emotion recognition is a notoriously challenging task, hindered by annotation cost and label ambiguity. In this work, we consider the task of learning embeddings for speech classification without access to any form of labelled audio. We base our approach on a simple hypothesis: that the emotional content of speech correlates with the facial expression of the speaker. By exploiting this relationship, we show that annotations of expression can be transferred from the visual domain (faces) to the speech domain (voices) through cross-modal distillation. We make the following contributions: (i) we develop a strong teacher network for facial emotion recognition that achieves the state of the art on a standard benchmark; (ii) we use the teacher to train a student, tabula rasa, to learn representations (embeddings) for speech emotion recognition without access to labelled audio data; and (iii) we show that the speech emotion embedding can be used for speech emotion recognition on external benchmark datasets. Code, models and data are available.

Related Papers

• → Types of Ambiguity(2006)4 cited
• → Finding and Using Ambiguity to Search for Innovation Opportunities(2018)2 cited
• On the Treatment of Intentional Ambiguity and Unintentional Ambiguity in English(2005)
• Study on the Pragmatic Value of English Intentional Ambiguity(2014)
• Theoretical Analysis of the Benchmark for Choosing Manipulative Instruments of Monetary Policies(2009)