Application of Mixed Reality (MR) with Gesture Recognition for Teaching and Training Repetitive Movements in Intangible Cultural Heritage Crafts

The preservation and transmission of Intangible Cultural Heritage (ICH) face significant challenges in today’s globalised world, mainly when teaching complex traditional handicraft techniques that involve repetitive and precise hand movements. This study explores the application of Mixed Reality (MR) technology combined with gesture recognition to enhance the teaching and training of these repetitive movements, focusing on the thread-winding process in Yunnan-style kite making as a case study. MR enables learners to interact with virtual materials in a simulated environment, while gesture recognition, implemented using Google’s MediaPipe, captures and evaluates hand movements in real time.By comparing learners’ gestures with the movements of skilled artisans, the MR system provides immediate feedback to correct technique and improve accuracy. The research demonstrates that MR-assisted gesture recognition significantly enhances the standardisation of repetitive movements, improves teaching quality, and reduces training time and material costs. Importantly, repetitive actions—essential to mastering traditional handcrafts—benefit from the immersive, interactive feedback that MR provides, helping learners refine their movements through continuous practice.This study contributes to the growing field of digital heritage education by showcasing how MR can modernise the transmission of traditional craft skills. In addition, it highlights the potential of gesture recognition technology in advancing the teaching and training of intricate motor skills in ICH practices. The experimental results suggest that MR-based systems could be valuable in preserving and promoting craftsmanship by offering scalable, cost-effective training solutions.