Leveraging body pose estimation for gesture recognition in human-robot interaction using synthetic data

Citations Over Time

Abstract

Effectively recognizing human gestures from variant viewpoints plays a fundamental role in the successful collaboration between humans and robots. Deep learning approaches have achieved promising performance in gesture recognition. However, they are usually data-hungry and require large-scale labeled data, which are not usually accessible in a practical setting. Synthetic data, on the other hand, can be easily obtained from simulators with fine-grained annotations and variant modalities. Existing state-of-the-art approaches have shown promising results using synthetic data, but there is still a large performance gap between the models trained on synthetic data and real data. To learn domain-invariant feature representations, we propose a novel approach which jointly takes RGB videos and 3D meshes as inputs to perform robust action recognition. We empirically validate our model on the RoCoG-v2 dataset, which consists of a variety of real and synthetic videos of gestures from the ground and air perspectives. We show that our model trained on synthetic data can outperform state-of-the-art models under the same training setting and models trained on real data.

Related Papers

• → 3D object pose estimation using stereo vision for object manipulation system(2017)21 cited
• → Articulated Human Body: 3D Pose Estimation Using a Single Camera(2010)3 cited
• Automatic markerless pose estimation for human in short sleeve or long sleeve attire(2008)
• An adaptable system for RGB-D based human body detection and pose estimation: Incorporating attached props(2016)
• → Real-Time Marker-Based Tracking and Pose Estimation for a Rotating Object Using High-Speed Vision(2022)