Slide
Centre Interdisciplinaire
de Recherche et d’Innovation
en Cybersécurité et Société
de Recherche et d’Innovation
en Cybersécurité et Société
1.
Joudeh, I. O.; Cretu, A. -M.; Bouchard, S.
Predicting the Arousal and Valence Values of Emotional States Using Learned, Predesigned, and Deep Visual Features † Article de journal
Dans: Sensors, vol. 24, no 13, 2024, ISSN: 14248220 (ISSN), (Publisher: Multidisciplinary Digital Publishing Institute (MDPI)).
Résumé | Liens | BibTeX | Étiquettes: adult, Affective interaction, Arousal, artificial neural network, Cognitive state, Cognitive/emotional state, Collaborative interaction, computer, Convolutional neural networks, correlation coefficient, Deep learning, emotion, Emotional state, Emotions, female, Forecasting, Helmet mounted displays, human, Humans, Learning algorithms, Learning systems, Long short-term memory, Machine learning, Machine-learning, male, Mean square error, Neural networks, physiology, Regression, Root mean squared errors, Video recording, virtual reality, Visual feature, visual features
@article{joudeh_predicting_2024,
title = {Predicting the Arousal and Valence Values of Emotional States Using Learned, Predesigned, and Deep Visual Features †},
author = {I. O. Joudeh and A. -M. Cretu and S. Bouchard},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-85198382238&doi=10.3390%2fs24134398&partnerID=40&md5=cefa8b2e2c044d02f99662af350007db},
doi = {10.3390/s24134398},
issn = {14248220 (ISSN)},
year = {2024},
date = {2024-01-01},
journal = {Sensors},
volume = {24},
number = {13},
abstract = {The cognitive state of a person can be categorized using the circumplex model of emotional states, a continuous model of two dimensions: arousal and valence. The purpose of this research is to select a machine learning model(s) to be integrated into a virtual reality (VR) system that runs cognitive remediation exercises for people with mental health disorders. As such, the prediction of emotional states is essential to customize treatments for those individuals. We exploit the Remote Collaborative and Affective Interactions (RECOLA) database to predict arousal and valence values using machine learning techniques. RECOLA includes audio, video, and physiological recordings of interactions between human participants. To allow learners to focus on the most relevant data, features are extracted from raw data. Such features can be predesigned, learned, or extracted implicitly using deep learners. Our previous work on video recordings focused on predesigned and learned visual features. In this paper, we extend our work onto deep visual features. Our deep visual features are extracted using the MobileNet-v2 convolutional neural network (CNN) that we previously trained on RECOLA’s video frames of full/half faces. As the final purpose of our work is to integrate our solution into a practical VR application using head-mounted displays, we experimented with half faces as a proof of concept. The extracted deep features were then used to predict arousal and valence values via optimizable ensemble regression. We also fused the extracted visual features with the predesigned visual features and predicted arousal and valence values using the combined feature set. In an attempt to enhance our prediction performance, we further fused the predictions of the optimizable ensemble model with the predictions of the MobileNet-v2 model. After decision fusion, we achieved a root mean squared error (RMSE) of 0.1140, a Pearson’s correlation coefficient (PCC) of 0.8000, and a concordance correlation coefficient (CCC) of 0.7868 on arousal predictions. We achieved an RMSE of 0.0790, a PCC of 0.7904, and a CCC of 0.7645 on valence predictions. © 2024 by the authors.},
note = {Publisher: Multidisciplinary Digital Publishing Institute (MDPI)},
keywords = {adult, Affective interaction, Arousal, artificial neural network, Cognitive state, Cognitive/emotional state, Collaborative interaction, computer, Convolutional neural networks, correlation coefficient, Deep learning, emotion, Emotional state, Emotions, female, Forecasting, Helmet mounted displays, human, Humans, Learning algorithms, Learning systems, Long short-term memory, Machine learning, Machine-learning, male, Mean square error, Neural networks, physiology, Regression, Root mean squared errors, Video recording, virtual reality, Visual feature, visual features},
pubstate = {published},
tppubtype = {article}
}
The cognitive state of a person can be categorized using the circumplex model of emotional states, a continuous model of two dimensions: arousal and valence. The purpose of this research is to select a machine learning model(s) to be integrated into a virtual reality (VR) system that runs cognitive remediation exercises for people with mental health disorders. As such, the prediction of emotional states is essential to customize treatments for those individuals. We exploit the Remote Collaborative and Affective Interactions (RECOLA) database to predict arousal and valence values using machine learning techniques. RECOLA includes audio, video, and physiological recordings of interactions between human participants. To allow learners to focus on the most relevant data, features are extracted from raw data. Such features can be predesigned, learned, or extracted implicitly using deep learners. Our previous work on video recordings focused on predesigned and learned visual features. In this paper, we extend our work onto deep visual features. Our deep visual features are extracted using the MobileNet-v2 convolutional neural network (CNN) that we previously trained on RECOLA’s video frames of full/half faces. As the final purpose of our work is to integrate our solution into a practical VR application using head-mounted displays, we experimented with half faces as a proof of concept. The extracted deep features were then used to predict arousal and valence values via optimizable ensemble regression. We also fused the extracted visual features with the predesigned visual features and predicted arousal and valence values using the combined feature set. In an attempt to enhance our prediction performance, we further fused the predictions of the optimizable ensemble model with the predictions of the MobileNet-v2 model. After decision fusion, we achieved a root mean squared error (RMSE) of 0.1140, a Pearson’s correlation coefficient (PCC) of 0.8000, and a concordance correlation coefficient (CCC) of 0.7868 on arousal predictions. We achieved an RMSE of 0.0790, a PCC of 0.7904, and a CCC of 0.7645 on valence predictions. © 2024 by the authors.
2.
Renaud, P.; Cusson, J. -F.; Bernier, S.; Mus, J. D. M.; Gourd, S. -P.; Bouchard, S.
Extracting perceptual and motor invariants using eye-tracking technologies in virtual immersions Article d'actes
Dans: Proceedings: HAVE 2002 - IEEE International Workshop on Haptic Virtual Environments and their Applications, p. 73–78, Institute of Electrical and Electronics Engineers Inc., 2002, ISBN: 0780376358 (ISBN); 978-078037635-9 (ISBN), (Journal Abbreviation: Proc.: HAVE - IEEE Int. Workshop Haptic Virtual Environ. Their Appl.).
Résumé | Liens | BibTeX | Étiquettes: Behavioral dynamics, Degrees of freedom (mechanics), Display devices, Eye movements, Eye tracking technologies, Head mounted displays, Head movements, Helmet mounted displays, Mathematical properties, Mechanics, Motor co-ordination, Position and orientations, Sensory perception, Tracking devices, virtual reality
@inproceedings{renaud_extracting_2002,
title = {Extracting perceptual and motor invariants using eye-tracking technologies in virtual immersions},
author = {P. Renaud and J. -F. Cusson and S. Bernier and J. D. M. Mus and S. -P. Gourd and S. Bouchard},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-84963544835&doi=10.1109%2fHAVE.2002.1106917&partnerID=40&md5=880d707663a5ed94337d041f2215a515},
doi = {10.1109/HAVE.2002.1106917},
isbn = {0780376358 (ISBN); 978-078037635-9 (ISBN)},
year = {2002},
date = {2002-01-01},
booktitle = {Proceedings: HAVE 2002 - IEEE International Workshop on Haptic Virtual Environments and their Applications},
pages = {73–78},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {From a head-mounted display, head and eye movements were recorded using tracking devices (magnetic and infrared) that render the 6 degrees-of-freedom associated with the position and orientation of head movements, and 2 degrees-of-freedom from one eye. We measured the continuous line of sight's deviation from a pre-selected area on a virtual stimulus. Some analyses of the mathematical properties of the emergent perceptual and motor patterns were done. These analyses help to understand how behavioral dynamics, and especially motor coordination underlying perception in virtual reality (VR), is organized. © 2002 IEEE.},
note = {Journal Abbreviation: Proc.: HAVE - IEEE Int. Workshop Haptic Virtual Environ. Their Appl.},
keywords = {Behavioral dynamics, Degrees of freedom (mechanics), Display devices, Eye movements, Eye tracking technologies, Head mounted displays, Head movements, Helmet mounted displays, Mathematical properties, Mechanics, Motor co-ordination, Position and orientations, Sensory perception, Tracking devices, virtual reality},
pubstate = {published},
tppubtype = {inproceedings}
}
From a head-mounted display, head and eye movements were recorded using tracking devices (magnetic and infrared) that render the 6 degrees-of-freedom associated with the position and orientation of head movements, and 2 degrees-of-freedom from one eye. We measured the continuous line of sight's deviation from a pre-selected area on a virtual stimulus. Some analyses of the mathematical properties of the emergent perceptual and motor patterns were done. These analyses help to understand how behavioral dynamics, and especially motor coordination underlying perception in virtual reality (VR), is organized. © 2002 IEEE.
3.
Bouchard, S.; Renaud, P.; Robillard, G.; St-Jacques, J.
Applications of virtual reality in clinical psychology: Illustrations with the treatment of anxiety disorders Article d'actes
Dans: Proceedings: HAVE 2002 - IEEE International Workshop on Haptic Virtual Environments and their Applications, p. 7–11, Institute of Electrical and Electronics Engineers Inc., 2002, ISBN: 0780376358 (ISBN); 978-078037635-9 (ISBN), (Journal Abbreviation: Proc.: HAVE - IEEE Int. Workshop Haptic Virtual Environ. Their Appl.).
Résumé | Liens | BibTeX | Étiquettes: 3D games, Clinical application, Clinical applications, clinical trial, Head mounted displays, Helmet mounted displays, High costs, Immersive environment, phobias, Scientific researches, Sensory perception, Surveys, virtual reality
@inproceedings{bouchard_applications_2002,
title = {Applications of virtual reality in clinical psychology: Illustrations with the treatment of anxiety disorders},
author = {S. Bouchard and P. Renaud and G. Robillard and J. St-Jacques},
url = {https://www.scopus.com/inward/record.uri?eid=2-s2.0-84963625772&doi=10.1109%2fHAVE.2002.1106906&partnerID=40&md5=8fa4558c35ab60c29d321a3a70bcbc4c},
doi = {10.1109/HAVE.2002.1106906},
isbn = {0780376358 (ISBN); 978-078037635-9 (ISBN)},
year = {2002},
date = {2002-01-01},
booktitle = {Proceedings: HAVE 2002 - IEEE International Workshop on Haptic Virtual Environments and their Applications},
pages = {7–11},
publisher = {Institute of Electrical and Electronics Engineers Inc.},
abstract = {The high cost of VR equipment and environments is a significant deterrent to the widespread dissemination of this technology from scientific research labs to the offices of clinical psychologists. An alternative is to use modified 3D games to create new and clinically useful immersive environments on head mounted displays. The aim of this paper is to summarize the first 18 months of work at the Cyberpsychology Lab. of the University of Quebec in Outaouais on the applications of virtual reality (VR) in the treatment of anxiety disorders. Three studies are summarized: a comparison between phobics and non-phobics immersed in VR environments, a clinical trial on the effectiveness of 3D games to treat phobias in adults and a clinical trial with children. Presence ratings are very high, as well as the graphic qualities and the flexibility of the programs. Results showed a dramatic improvement on general questionnaires and on measures of overt approach/avoidance of phobic stimuli. © 2002 IEEE.},
note = {Journal Abbreviation: Proc.: HAVE - IEEE Int. Workshop Haptic Virtual Environ. Their Appl.},
keywords = {3D games, Clinical application, Clinical applications, clinical trial, Head mounted displays, Helmet mounted displays, High costs, Immersive environment, phobias, Scientific researches, Sensory perception, Surveys, virtual reality},
pubstate = {published},
tppubtype = {inproceedings}
}
The high cost of VR equipment and environments is a significant deterrent to the widespread dissemination of this technology from scientific research labs to the offices of clinical psychologists. An alternative is to use modified 3D games to create new and clinically useful immersive environments on head mounted displays. The aim of this paper is to summarize the first 18 months of work at the Cyberpsychology Lab. of the University of Quebec in Outaouais on the applications of virtual reality (VR) in the treatment of anxiety disorders. Three studies are summarized: a comparison between phobics and non-phobics immersed in VR environments, a clinical trial on the effectiveness of 3D games to treat phobias in adults and a clinical trial with children. Presence ratings are very high, as well as the graphic qualities and the flexibility of the programs. Results showed a dramatic improvement on general questionnaires and on measures of overt approach/avoidance of phobic stimuli. © 2002 IEEE.