The classification of selected facial expressions using single-channel electromyographic signals through artificial neural networks and decision trees
Subject Areas : Biomedical EngineeringNiloofar Shojaeian 1 , babak Rezae Afshar 2
1 - Department of Biomedical Engineering Group, Bioelectrics, Islamic Azad University, Central Tehran Branch, Tehran, Iran
2 - Head of Cognetive science lab ,incubator center of mohaghegh ardebili univrsity.
Keywords: Facial expressions, electromyography, artificial neural network, decision tree, principal component analysis,
Abstract :
This study examines various facial expressions using single-channel electromyography with minimal computational complexity for real-time applications. The electromyography data of facial muscles is derived from the electrical potential generated by different facial expressions, which can have varying values for each expression. In this study, electromyography data were recorded from the facial muscles of six female participants with a mean age of 25±5 years. After applying a fourth-order Butterworth filter, the data were divided into 10-second windows, and 12 features were extracted in both time and frequency domains. To reduce computational load, Principal Component Analysis (PCA) was utilized for feature reduction, while classification of the data was performed using artificial neural networks and decision trees. After applying the filters and extracting features, the data were classified using multilayer perceptron classifiers with an accuracy of 73.57%, decision trees with an accuracy of 95.16%, MPL-PCA with an accuracy of 65.20%, and DT-PCA with an accuracy of 96.33%. Challenges in this study include the small number of participants and the single-channel nature of the data. Future studies could combine facial muscle data.
V. Kehri and R. Awale, "A facial EMG data analysis for emotion classification based on spectral kurtogram and CNN," International Journal of digital signals and smart systems, vol. 4, pp. 50-63, 2020. https://doi.org/10.1504/IJDSSS.2020.106072
[2] K.-H. S. Chen and R. Chen, "Principles of electrophysiological assessments for movement disorders," Journal of movement disorders, vol. 13, p. 27, 2020. https://doi.org/10.14802/jmd.19064
[3] C. L. Kok, C. K. Ho, F. K. Tan, and Y. Y. Koh, "Machine learning-based feature extraction and classification of emg signals for intuitive prosthetic control," Applied Sciences, vol. 14, p. 5784, 2024. https://doi.org/10.3390/app14135784
[4] M. Gjoreski, I. Kiprijanovska, S. Stankoski, I. Mavridou, M. J. Broulidakis, H. Gjoreski, et al., "Facial EMG sensing for monitoring affect using a wearable device," Scientific reports, vol. 12, p. 16876, 2022.
[5] J. Zhang, X. Xie, G. Peng, L. Liu, H. Yang, R. Guo, et al., "A Real-Time and Privacy-Preserving Facial Expression Recognition System Using an AI-Powered Microcontroller," Electronics, vol. 13, p. 2791, 2024. https://doi.org/10.3390/electronics13142791
[6] S. A. Hussain and A. S. A. Al Balushi, "A real time face emotion classification and recognition using deep learning model," in Journal of physics: Conference series, 2020, p. 012087. 10.1088/1742-6596/1432/1/012087
[7] E. S. Agung, A. P. Rifai, and T. Wijayanto, "Image-based facial emotion recognition using convolutional neural network on emognition dataset," Scientific reports, vol. 14, p. 14429, 2024.
[8] J. Wei, G. Hu, X. Yang, A. T. Luu, and Y. Dong, "Learning facial expression and body gesture visual information for video emotion recognition," Expert Systems with Applications, vol. 237, p. 121419, 2024. https://doi.org/10.1016/j.eswa.2023.121419
[9] J. Basavaiah, A. A. Anthony, N. K. HN, and C. M. Patil, "Facial Emotion Recognition: A Review on State-of-the-art Techniques," in 2024 4th International Conference on Data Engineering and Communication Systems (ICDECS), 2024, pp. 1-6.
[10] S. Casaccia, E. J. Sirevaag, M. G. Frank, J. A. O’Sullivan, L. Scalise, and J. W. Rohrbaugh, "Facial muscle activity: High-sensitivity noncontact measurement using laser Doppler vibrometry," IEEE Transactions on Instrumentation and Measurement, vol. 70, pp. 1-10, 2021. https://doi.org/10.1109/TIM.2021.3060564
[11] Q. Zhu, G. Lu, and J. Yan, "Valence-arousal model based emotion recognition using EEG, peripheral physiological signals and facial expression," in Proceedings of the 4th International Conference on Machine Learning and Soft Computing, 2020, pp. 81-85. https://doi.org/10.1145/3380688.3380694
[12] A. Kelati, J. Plosila, and H. Tenhunen, "Machine learning for semg facial feature characterization," in 2019 Signal Processing: Algorithms, Architectures, Arrangements, and Applications (SPA), 2019, pp. 169-174. https://doi.org/10.23919/SPA.2019.8936818
[13] S. S. Blom, H. Aarts, and G. R. Semin, "Lateralization of facial emotion processing and facial mimicry," Laterality, vol. 25, pp. 259-274, 2020. https://doi.org/10.1080/1357650X.2019.1657127
[1] V. Kehri and R. Awale, "A facial EMG data analysis for emotion classification based on spectral kurtogram and CNN," International Journal of digital signals and smart systems, vol. 4, pp. 50-63, 2020. https://doi.org/10.1504/IJDSSS.2020.106072
[2] K.-H. S. Chen and R. Chen, "Principles of electrophysiological assessments for movement disorders," Journal of movement disorders, vol. 13, p. 27, 2020. https://doi.org/10.14802/jmd.19064
[3] C. L. Kok, C. K. Ho, F. K. Tan, and Y. Y. Koh, "Machine learning-based feature extraction and classification of emg signals for intuitive prosthetic control," Applied Sciences, vol. 14, p. 5784, 2024. https://doi.org/10.3390/app14135784
[4] M. Gjoreski, I. Kiprijanovska, S. Stankoski, I. Mavridou, M. J. Broulidakis, H. Gjoreski, et al., "Facial EMG sensing for monitoring affect using a wearable device," Scientific reports, vol. 12, p. 16876, 2022.
[5] J. Zhang, X. Xie, G. Peng, L. Liu, H. Yang, R. Guo, et al., "A Real-Time and Privacy-Preserving Facial Expression Recognition System Using an AI-Powered Microcontroller," Electronics, vol. 13, p. 2791, 2024. https://doi.org/10.3390/electronics13142791
[6] S. A. Hussain and A. S. A. Al Balushi, "A real time face emotion classification and recognition using deep learning model," in Journal of physics: Conference series, 2020, p. 012087. 10.1088/1742-6596/1432/1/012087
[7] E. S. Agung, A. P. Rifai, and T. Wijayanto, "Image-based facial emotion recognition using convolutional neural network on emognition dataset," Scientific reports, vol. 14, p. 14429, 2024.
[8] J. Wei, G. Hu, X. Yang, A. T. Luu, and Y. Dong, "Learning facial expression and body gesture visual information for video emotion recognition," Expert Systems with Applications, vol. 237, p. 121419, 2024. https://doi.org/10.1016/j.eswa.2023.121419
[9] J. Basavaiah, A. A. Anthony, N. K. HN, and C. M. Patil, "Facial Emotion Recognition: A Review on State-of-the-art Techniques," in 2024 4th International Conference on Data Engineering and Communication Systems (ICDECS), 2024, pp. 1-6.
[10] S. Casaccia, E. J. Sirevaag, M. G. Frank, J. A. O’Sullivan, L. Scalise, and J. W. Rohrbaugh, "Facial muscle activity: High-sensitivity noncontact measurement using laser Doppler vibrometry," IEEE Transactions on Instrumentation and Measurement, vol. 70, pp. 1-10, 2021. https://doi.org/10.1109/TIM.2021.3060564
[11] Q. Zhu, G. Lu, and J. Yan, "Valence-arousal model based emotion recognition using EEG, peripheral physiological signals and facial expression," in Proceedings of the 4th International Conference on Machine Learning and Soft Computing, 2020, pp. 81-85. https://doi.org/10.1145/3380688.3380694
[12] A. Kelati, J. Plosila, and H. Tenhunen, "Machine learning for semg facial feature characterization," in 2019 Signal Processing: Algorithms, Architectures, Arrangements, and Applications (SPA), 2019, pp. 169-174. https://doi.org/10.23919/SPA.2019.8936818
[13] S. S. Blom, H. Aarts, and G. R. Semin, "Lateralization of facial emotion processing and facial mimicry," Laterality, vol. 25, pp. 259-274, 2020. https://doi.org/10.1080/1357650X.2019.1657127
[14] M. N. Kyranides, M. Petridou, H. A. Gokani, S. Hill, and K. A. Fanti, "Reading and reacting to faces, the effect of facial mimicry in improving facial emotion recognition in individuals with antisocial behavior and psychopathic traits," Current Psychology, vol. 42, pp. 14306-14319, 2023.
[15] A. González-Rodríguez, Á. García-Pérez, M. Godoy-Giménez, P. Sayans-Jiménez, F. Cañadas, and A. F. Estévez, "The role of the differential outcomes procedure and schizotypy in the recognition of dynamic facial expressions of emotions," Scientific Reports, vol. 14, p. 2322, 2024.
[16] L. O. LUNDQVIST, "Facial EMG reactions to facial expressions: A case of facial emotional contagion?," Scandinavian journal of psychology, vol. 36, pp. 130-141, 1995. https://doi.org/10.1111/j.1467-9450.1995.tb00974.x
[17] B. 't Hart, M. E. Struiksma, A. van Boxtel, and J. J. Van Berkum, "Emotion in stories: Facial EMG evidence for both mental simulation and moral evaluation," Frontiers in psychology, vol. 9, p. 613, 2018. https://doi.org/10.3389/fpsyg.2018.00613
[18] J. J. van Berkum, M. Struiksma, and B. t Hart, "Using facial EMG to track emotion during language comprehension: past, present, and future," in Language Electrified, ed: Springer, 2023, pp. 687-729.
[19] J. Chen, T. Ro, and Z. Zhu, "Emotion recognition with audio, video, EEG, and EMG: a dataset and baseline approaches," IEEE Access, vol. 10, pp. 13229-13242, 2022.
[20] C.-T. Hsu and W. Sato, "Electromyographic validation of spontaneous facial mimicry detection using automated facial action coding," Sensors, vol. 23, p. 9076, 2023.
[21] G. Lombardi, M. Gerbella, M. Marchi, A. Sciutti, G. Rizzolatti, and G. Di Cesare, "Investigating form and content of emotional and non-emotional laughing," Cerebral Cortex, vol. 33, pp. 4164-4172, 2023. https://doi.org/10.1093/cercor/bhac334
[22] L. Franz, C. de Filippis, A. Daloiso, E. Biancoli, F. P. Iannacone, D. Cazzador, et al., "Facial surface electromyography: A systematic review on the state of the art and current perspectives," American Journal of Otolaryngology, vol. 45, p. 104041, 2024. https://doi.org/10.1016/j.amjoto.2023.104041
[23] A. van Boxtel, "Electromyographic (EMG) responses of facial muscles during language processing," in Language Electrified: Principles, Methods, and Future Perspectives of Investigation, ed: Springer, 2023, pp. 367-385.
[24] C. J. Doelman and J. A. Rijken, "Yawning and airway physiology: a scoping review and novel hypothesis," Sleep and Breathing, vol. 26, pp. 1561-1572, 2022.
[25] B. Merzoug, M. Ouslim, L. Mostefai, and M. Benouis, "Evaluation of dimensionality reduction using PCA on EMG-based signal pattern classification," Engineering Proceedings, vol. 14, p. 23, 2022. https://doi.org/10.3390/engproc2022014023
[26] T. S. Murotzhonovich, "Introduction to Artificial Neural Networks," Web of Synergy: International Interdisciplinary Research Journal, vol. 2, pp. 198-202, 2023. 10.4018/979-8-3693-7250-0.ch016
[27] A. K. Ali and E. Erçelebi, "Automatic modulation classification using different neural network and PCA combinations," Expert Systems with Applications, vol. 178, p. 114931, 2021. https://doi.org/10.1016/j.eswa.2021.114931
[28] F. Vargas, D. Lettnin, M. C. F. de Castro, and M. Macarthy, "Electrocardiogram pattern recognition by means of MLP network and PCA: A case study on equal amount of input signal types," in VII Brazilian Symposium on Neural Networks, 2002. SBRN 2002. Proceedings., 2002, pp. 200-205. https://doi.org/10.1109/SBRN.2002.1181474
[29] L. Shao, "Facial movements recognition using multichannel EMG signals," in 2019 IEEE Fourth International Conference on Data Science in Cyberspace (DSC), 2019, pp. 561-566.
[30] J. Zhang, S. Huang, J. Li, Y. Wang, Z. Dong, and S.-J. Wang, "A Perifacial EMG Acquisition System for Facial-Muscle-Movement Recognition," Sensors, vol. 23, p. 8758, 2023.
[31] S. A. Mithbavkar and M. S. Shah, "Recognition of emotion through facial expressions using EMG signal," in 2019 international conference on nascent technologies in engineering (ICNTE), 2019, pp. 1-6.
[32] L. Kulke, D. Feyerabend, and A. Schacht, "A comparison of the Affectiva iMotions Facial Expression Analysis Software with EMG for identifying facial expressions of emotion," Frontiers in psychology, vol. 11, p. 329, 2020.
