Real Time Piano Finger Recognition Using Convolutional Neural Networks and Gated Recurrent Units
Abstract
Standardized piano fingering practice not only concerns the stability of performance, but also directly affects the richness and coherence of musical expression. However, due to limitations in teacher resources and teaching time, existing piano teaching has significant shortcomings in personalized guidance and immediate feedback on errors. Therefore, this article proposes a computer vision based piano finger recognition and training system, which integrates convolutional neural networks (CNN) for hand static feature extraction, gated recurrent units (GRU) for action time series modeling, and introduces spatial attention and temporal attention mechanisms to improve finger recognition accuracy and dynamic response capability. The experiment was conducted based on a self built dataset of over 130000 piano performance images. The system outperformed existing methods in key indicators such as finger gesture recognition accuracy (89.8%), macro average F1 value (90.2%), and feedback response delay (0.47 seconds), especially in complex action recognition and real time feedback. The research results provide feasible technical support for the piano intelligent teaching system and have good application and promotion value.
Full Text:
PDFReferences
Kapuscinski T , Majcher M .Vision-based fingerspelling recognition for an assistive technology system[J].Procedia Computer Science,2024,237:469-476.
Yiqun C .Interactive piano training using augmented reality and the Internet of Things[J].Education and Information Technologies,2022,28(6):6373-6389.
Ji C ,Wang D ,Wang H .An investigation on the application of deep reinforcement learning in piano playing technique training[J].Applied Mathematics and Nonlinear Sciences,2024,9(1):3135
Zhao X ,Wang Y ,Cai X .A ResNet-Based Audio-Visual Fusion Model for Piano Skill Evaluation[J].Applied Sciences,2023,13(13):7431
Ruan W .Increasing student motivation to learn the piano using modern digital technologies: independent piano learning with the soft Mozart app[J].Current Psychology, 2024, 43(44):33998-34008.
lara Schumann,The Technological Priestess: The Piano Recital, Photography, and Clara Schumann[J].Nineteenth-Century Music Review, 2024, 21(1):76-109.
Holzer D .A Piano for Visuals: Affordances of Scan Processing Instruments[J].Leonardo,2024,57(6):613-621.
YunDan Z ,Tian T ,Ai Z .Training Strategy of Music Expression in Piano Teaching and Performance by Intelligent Multimedia Technology[J].International Transactions on Electrical Energy Systems,2022,(2022):1
You W .Research on Rhythm Training of Double Piano Playing[J].Art and Performance Letters,2023,(4):1
Programming S .RETRACTION: Design and Application of the Piano Teaching System Integrating Videos and Images[J].Scientific Programming, 2024, (2024).1058-9244
Jingle H .Analysis of Different Keying Modes and Brain Sound Image Establishment Mode in Piano Playing[J].Wireless Communications and Mobile Computing,2022,(2022)1530-8669
Yu Linna,and Luo Zhifan."The Use of Artificial Intelligence Combined with Wireless Network in Piano Music Teaching."Wireless Communications and Mobile Computing 2022.(2022):1530-8669
Mathematical A C M I M .Retracted: Research on Classroom Online Teaching Model of “Learning” Wisdom Music on Wireless Network under the Background of Artificial Intelligence[J].Computational and Mathematical Methods in Medicine,2022,(2022):9782489-9782489.
Lin J , Ding B , Song Z ,et al.A Model of Multi-Finger Coordination in Keystroke Movement[J].Sensors, 2024, 24(4):17.
Takehara Y , Kitano K , Ito A ,et al.Motion analysis during piano playing using inertial sensors[J].The Proceedings of Mechanical Engineering Congress, Japan, 2022.（162）6.
Kanami I ,Tatsunori W ,Takayuki H , et al.Higher synchronization stability with piano experience: relationship with finger and presentation modality[J].Journal of Physiological Anthropology,2023,42(1):10-10.
Gao W ,Zhang S ,Zhang N , et al.Generating Fingerings for Piano Music with Model-Based Reinforcement Learning[J].Applied Sciences,2023,13(20):11321
Loo F Y , Chai K E , Loo F C ,et al.Exploring synergy in a mobile learning model for piano playing ornaments exercise with local musical heritage[J].International Journal of Music Education, 2022, 40(3):12.
Zhi Q .Feature Extraction Method of Piano Performance Technique Based on Recurrent Neural Network[J].International Journal of Gaming and Computer-Mediated Simulations (IJGCMS),2022,14(2):1-14.
Xin G ,Haoyue Z ,Qiang L .Estimation of playable piano fingering by pitch-difference fingering match model[J].EURASIP Journal on Audio, Speech, and Music Processing,2022,2022(1):1
Wang Q , Liu M , Bao C ,et al.Harmonic-Aware Frequency and Time Attention for Automatic Piano Transcription[J].Audio, Speech, and Language Processing, IEEE/ACM Transactions on, 2024, 32(000)3492- 3506
Huang H H , Lin Y B .IoT-Based Piano Playing Robot[J].IEEE Internet of Things Magazine, 2024(6):7.
Wang Y , Pang B , Zhai M .Research on piano online teaching instrument based on spherical off-axis four-reflective optical structure[J].Proceedings of SPIE, 2023, 12507(000):9.
Alghazali N O S , Musa T A .A comparative analysis of the energy dissipation efficiency of various piano key weir types[J].Open Engineering, 2024, 14(1):144-9.
Xu Q ,Li M ,Chen G , et al.Design of a Wearable Exoskeleton Piano Practice Aid Based on Multi-Domain Mapping and Top-Down Process Model[J].Biomimetics,2024,10(1):15-15.
Hao J .Online piano learning game design method: Piano music style recognition based on CRNNH[J].Entertainment Computing,2024,50(100)645.
DOI: https://doi.org/10.31449/inf.v49i6.9650
This work is licensed under a Creative Commons Attribution 3.0 License.