Speech Recognition with LLMs Adapted to Disordered Speech Using Reinforcement Learning

Citations Over TimeTop 10% of 2025 papers

Abstract

We introduce a large language model (LLM) capable of processing speech inputs and show that tuning it further with reinforcement learning on human preference (RLHF) enables it to adapt better to disordered speech than traditional fine-tuning. Our method replaces low-frequency text tokens in an LLM’s vocabulary with audio tokens and enables the model to recognize speech by fine-tuning it on speech with transcripts. We then use RL with rewards based on syntactic and semantic accuracy measures generalizing the LLM further to recognize disordered speech. While the resulting LLM does not outperform existing systems for speech recognition we find that tuning with reinforcement learning using custom rewards leads to substantially better performance than supervised fine-tuning of the language model, specifically when adapting to speech in a different setting. This presents a compelling alternative tuning strategy for speech recognition using large language models.

Related Papers

• → CHOICES BETWEEN POSITIVE AND NEGATIVE REINFORCEMENT DURING TREATMENT FOR ESCAPE‐MAINTAINED BEHAVIOR(2001)78 cited
• → Review on Reinforcement-concrete Bonded Anchorage(2022)1 cited
• → REINFORCEMENT OF INHIBITION(1983)9 cited
• → Variable Ratio Reinforcement and Differential Reinforcement(1974)
• → The Effect of Vicarious Reinforcement on Inappropriate Behavior in an Elementary School Classroom(1975)