Talking With Hands 16.2M: A Large-Scale Dataset of Synchronized Body-Finger Motion and Audio for Conversational Motion Analysis and Synthesis

Citations Over TimeTop 10% of 2019 papers

Abstract

We present a 16.2-million frame (50-hour) multimodal dataset of two-person face-to-face spontaneous conversations. Our dataset features synchronized body and finger motion as well as audio data. To the best of our knowledge, it represents the largest motion capture and audio dataset of natural conversations to date. The statistical analysis verifies strong intraperson and interperson covariance of arm, hand, and speech features, potentially enabling new directions on data-driven social behavior analysis, prediction, and synthesis. As an illustration, we propose a novel real-time finger motion synthesis method: a temporal neural network innovatively trained with an inverse kinematics (IK) loss, which adds skeletal structural information to the generative model. Our qualitative user study shows that the finger motion generated by our method is perceived as natural and conversation enhancing, while the quantitative ablation study demonstrates the effectiveness of IK loss.

Related Papers

• → Kinematic analysis of sit-to-stand assistive device for the elderly and disabled(2011)23 cited
• → Comparative Analysis of Biomechanical Variables in Marker-based and Markerless Motion Capture Systems(2023)5 cited
• → Evaluation of action sport camera optical motion capture system for 3D gait analysis(2021)4 cited
• → Kinematic Motion Analysis with Volumetric Motion Capture(2022)2 cited
• → Walking Gait Analysis: Kinovea versus Motion Capture System(2022)3 cited