Deep Emotion Recognition from Facial IPPG Signals: A Contactless Framework Using Transformer-Based Temporal Modeling

Subject Areas : International Journal of Smart Electrical Engineering

Mahnam Mirzaei ¹ , Mahdi Azarnoosh ^{2 *} , Hamidreza Kobravi ³

1 - Department of Biomedical Engineering, Ma.C., Islamic Azad University, Mashhad, Iran
2 - Department of Biomedical Engineering, Ma.C., Islamic Azad University, Mashhad, Iran
3 - Department of Biomedical Engineering, Ma.C., Islamic Azad University, Mashhad, Iran

Received: 2025-08-01 Accepted : 2025-10-10 Published : 2025-07-24

Keywords: Imaging Photoplethysmography, Emotion Recognition, Deep Learning, Facial Video Analysis, Non-Contact Monitoring,

Abstract :

Abstract— This study presents a contactless framework for deep emotion recognition using imaging photoplethysmography signals extracted from facial videos. Data were collected from 32 participants (16 males, 16 females, aged 20–35) using a 4K RGB webcam under ambient lighting conditions, while emotional states were induced using standardized stimuli from the DEAP, DREAMER, and LUMED-EmoStim (2024) databases. Facial landmarks were detected via MediaPipe, and a region of interest was defined on the upper cheek to extract green-channel-based IPPG signals, which were processed using adaptive filtering and bandpass filtering to isolate physiological components. Time and frequency domain features—including heart rate, pulse rate variability, signal entropy, and waveform statistics—were extracted from 10-second windows. Three deep learning models—Transformer, Conformer, and BiLSTM—were implemented to classify eight target emotions: Neutral, Happy, Surprised, Fearful, Angry, Disgusted, Sad, and Excited. Evaluation was conducted under both subject-dependent and subject-independent conditions using precision, recall, F1-score, and accuracy metrics. Results showed that all models achieved competitive performance (F1-score > 86%), with BiLSTM slightly outperforming others (F1 = 87.89%). While the Conformer excelled in capturing high-temporal-variability emotions like Fearful, the Transformer demonstrated stronger generalizability across subjects. Statistical analysis (ANOVA, p > 0.05) revealed no significant difference among models, indicating the robustness of the proposed pipeline. These findings highlight the potential of IPPG-based, non-contact emotion recognition systems for applications in telehealth, mental health monitoring, and affective computing.

References:

[1] Zhang R, Deng H, Xiao X. The insular cortex: an interface between sensation, emotion and cognition. Neuroscience Bulletin. 2024 Nov;40(11):1763-73.
[2] Maroju PK, Bhattacharya P. Understanding emotional intelligence: The heart of human-centered technology. InHumanizing Technology With Emotional Intelligence 2025 (pp. 1-18). IGI Global Scientific Publishing.
[3] Thirunagalingam A, Whig P. Emotional AI integrating human feelings in machine learning. InHumanizing Technology With Emotional Intelligence 2025 (pp. 19-32). IGI Global Scientific Publishing.
[4] Jin H, Qi C, Chen Z. Affective computing for healthcare: Recent trends, applications, challenges, and beyond. Emotional Intelligence. 2024 Feb 21:3.
[5] Mousavi SA, Tahami E, Bidaki MZ. The Effect of Using Virtual Reality Games on Health and Fitness. Journal of Computer & Robotics. 2023 Oct 1;17(1):17-26.
[6] Nawaz AH, Shahzad R, Ilyas S, Javed S. Emotion-Aware AI system in Education Supporting Student Mental Health and Learning Outcomes. The Critical Review of Social Sciences Studies. 2025 Jul 15;3(3):487-504.
[7] Xu X, Fu C, Camacho D, Park JH, Chen J. Internet of things for emotion care: Advances, applications, and challenges. Cognitive Computation. 2024 Nov;16(6):2812-32.
[8] Yadav G, Bokhari MU, Alzahrani SI, Alam S, Shuaib M. Emotion-aware ensemble learning (EAEL): revolutionizing Mental Health diagnosis of corporate professionals via Intelligent Integration of Multi-modal Data sources and ensemble techniques. IEEE Access. 2025 Jan 13.
[9] Nazari J, Nosheri AG, Mousavi SA. Hand Movements Detection Using EMG Signals for Human-Computer Interface and convolution neural network. In2024 20th CSI International Symposium on Artificial Intelligence and Signal Processing (AISP) 2024 Feb 21 (pp. 1-5). IEEE.
[10] Zhang J, Chen W. A Decade of Music Emotion Computing: A Bibliometric Analysis of Trends, Interdisciplinary Collaboration, and Applications. Education for Information. 2025 Aug;41(3):227-55.
[11] Faria DR, Godkin AL, da Silva Ayrosa PP. Advancing Emotionally Aware Child–Robot Interaction with Biophysical Data and Insight-Driven Affective Computing. Sensors. 2025 Feb 14;25(4):1161.
[12] Faria DR, Godkin AL, da Silva Ayrosa PP. Advancing Emotionally Aware Child–Robot Interaction with Biophysical Data and Insight-