mSLAM: Massively multilingual joint pre-training for speech and text

Citations Over Time

Abstract

We present mSLAM, a multilingual Speech and LAnguage Model that learns cross-lingual cross-modal representations of speech and text by pre-training jointly on large amounts of unlabeled speech and text in multiple languages. mSLAM combines w2v-BERT pre-training on speech with SpanBERT pre-training on character-level text, along with Connectionist Temporal Classification (CTC) losses on paired speech and transcript data, to learn a single model capable of learning from and representing both speech and text signals in a shared representation space. We evaluate mSLAM on several downstream speech understanding tasks and find that joint pre-training with text improves quality on speech translation, speech intent classification and speech language-ID while being competitive on multilingual ASR, when compared against speech-only pre-training. Our speech translation model demonstrates zero-shot text translation without seeing any text translation data, providing evidence for cross-modal alignment of representations. mSLAM also benefits from multi-modal fine-tuning, further improving the quality of speech translation by directly leveraging text translation data during the fine-tuning process. Our empirical analysis highlights several opportunities and challenges arising from large-scale multimodal pre-training, suggesting directions for future research.

Related Papers

• → The Kyoto Speech-to-Speech Translation System for IWSLT 2023(2023)2 cited
• Ogmios: The UPC Text-to-Speech synthesis system for Spoken Translation(2006)
• → DARIJA-C: towards a Moroccan DARIJA Speech recognition and speech-to-text Translation Corpus(2023)5 cited
• → Development of text and speech corpus for an Indonesian speech-to-speech translation system(2017)9 cited
• → A System Design of English Speech Synthesis(2021)