Am I Me or You? State-of-the-Art Dialogue Models Cannot Maintain an Identity
Citations Over TimeTop 14% of 2022 papers
Abstract
State-of-the-art dialogue models still often stumble with regards to factual accuracy and self-contradiction. Anecdotally, they have been observed to fail to maintain character identity throughout discourse; and more specifically, may take on the role of their interlocutor. In this work we formalize and quantify this deficiency, and show experimentally through human evaluations that this is indeed a problem. In contrast, we show that discriminative models trained specifically to recognize who is speaking can perform well; and further, these can be used as automated metrics. Finally, we evaluate a wide variety of mitigation methods, including changes to model architecture, training protocol, and decoding strategy. Our best models reduce mistaken identity issues by nearly 65% according to human annotators, while simultaneously improving engagingness. Despite these results, we find that maintaining character identity still remains a challenging problem.
Related Papers
- → The Principal Contradiction and the Principal Aspect of a Contradiction(1977)3 cited
- → On the Concept "Contradiction" in Dialectic(1968)
- A Discussion on Two Issues of Contradiction(2001)
- Another Probe into Contradiction in English(2002)
- Deng Xiaoping'S Development of Contradiction Theory(2006)