AE-LSTM-Based Multimodal Sensing System for Real-Time Monitoring of Children’s Play Behavior on Edge Devices

Abstract

Children’s play is a fundamental activity that supports emotional, cognitive, and social development. However, capturing and analyzing play behavior in real time is challenging due to its spontaneous, multimodal, and dynamic nature. Traditional observation methods are time-consuming, subjective, and lack real-time responsiveness. This research aims to design and implement a multimodal sensing and feedback platform that leverages edge computing and real-time Artificial Intelligence (AI) to monitor, interpret, and support children’s play behavior. The platform collects multimodal play behavior datasets from various sensors, including action and posture recognition, microphones for speech and voice tone analysis, motion sensors to track physical activity, and wearable devices. An Autoencoder-based Long Short-Term Memory (AE-LSTM) network is used to analyze behavior in real time. Feature extraction is performed using a lightweight ResNet model to extract features. Data is pre-processed using Kalman filtering and normalization techniques to reduce noise and improve consistency. The entire system is deployed on edge devices to ensure low-latency processing, local storage, and privacy preservation. The system also provides real-time feedback through visual and haptic cues to enhance engagement. Implemented in Python, experiments have demonstrated that the proposed AE-LSTM model outperforms baseline architectures like LSTM, GRU, and BiLSTM+Attention, and the proposed model achieves higher results according to the F1-score (0.959), accuracy (0.975), recall (0.964), and precision (0.968). These findings offer robust performance in naturalistic settings and provide valuable applications for educators, therapists, and researchers who intend to support and understand child development through intelligent, responsive play environments.