Sensorimotor Regularities as Alignment between Humans and Large Language Models

Abstract

Large Language Models (LLMs) do not construct conceptual representations in ways that align with human cognition, posing risks for human-AI interaction. While LLMs solely rely on linguistic distributional knowledge, humans leverage both linguistic and sensorimotor knowledge. To systematically assess human-LLM alignment in concept representations, we propose a novel evaluation framework based on sensorimotor regularities, operationalized as image schemas—multimodal gestalts derived from repeated sensorimotor experiences. Investigating linguistic manifestations of such schemas, we systematically identify human-LLM alignments and misalignments in the encoding of sensorimotor regularities. Results indicate that three contemporary disembodied LLMs encode highly human-like sensorimotor gestalts. However, these models exhibit reduced alignment when mapping such gestalts to concepts, and they do not systematically combine these gestalts in ways consistent with human patterns. We identify each LLM’s misalignments with human patterns in image schema distribution, conceptual associations, and image schema co-occurrences. Building on these findings, we augment gpt-4-1106 with targeted sensorimotor priors derived from its identified misalignments with human patterns. In a downstream user study, this augmentation yields sentence continuations rated by humans as significantly more conceptually clear, contextually contingent, and human-like than baseline outputs. Our work establishes a foundation for evaluating and improving human-LLM alignment at the conceptual level.