Automated Classification of User Exercise Poses in Virtual Reality using Machine Learning-Based Human Pose Estimation

Abstract

Human Pose Estimation (HPE) using machine learning presents significant potential for objectively analyzing user movements within Virtual Reality (VR) environments, particularly for applications involving guided physical exercises or interactive tasks. Analyzing 2D video footage from VR sessions for HPE offers a computationally efficient approach for tracking user movements. However, accuracy can be compromised by various factors, including occlusions from clothing or the VR headset itself. This study develops and evaluates a robust HPE system specifically designed to accurately classify predefined poses performed by users within such VR environments. The proposed system utilizes MediaPipe for corporal landmark extraction from user images (for training) and employs a Stacking Classifier ensemble, with XGBClassifier as the meta-learner, to classify six key exercise poses. Key results demonstrate an overall pose classification accuracy of 94% on a dedicated test set, with certain poses like `Butterfly Hug' and `Arm' achieving 100% accuracy. These findings highlight the system’s potential to provide reliable, quantitative assessments of user adherence to prescribed movements in immersive interactive environments, offering valuable data for applications requiring objective analysis of physical performance in VR.