Researchers at the University of Skövde have “trained” a neural network that can detect the emotional state of a person playing a computer game with the help of an ordinary web camera. This might open up for a future where computer games adjust themselves to the emotional state of the gamer.
Imagine yourself playing a computer game. While you are playing, a camera on the computer records a video of your face. Solely by analyzing the video, the computer detects whether you find the game enjoyable, boring, easy or hard. The game will then adjust itself to optimize your gaming experience.
Computer games available today use only facial analysis or inconvenient physical sensors to have the ability to adjust itself to the actual feelings of the gamer. But due to research conducted at the University of Skövde this might change. During the work with his dissertation, PhD-student Fernando Bevilacqua has constructed a neural network that can detect the emotional state of someone playing a computer game only by analyzing a video of the gamer’s face.
Trained with Super Mario Bros
A neural network is kind of AI-software vaguely inspired by the biological neural networks that constitute the human brain. Fernando Bevilacqua “trained” his neural network and tested it by letting a number of people play a modified version of the game Super Mario Bros. An ordinary web camera was recording a video of the gamer’s face while he or she was playing the game. The video was then put through a process known as remote photoplethysmography where the gamer’s heart rate can be detected. Information about the gamer’s facial expressions was also retrieved from the video and fed into the neural network. From this information the neural network was able to accurately detect the gamer’s emotional state in more than 60 percent of the cases.
– During my PhD-work I have focused on gaming and I have used a computer game to calibrate the neural network. But I think this technique could be used within other fields as well. Perhaps we could use it in cars to tell the driver when he or she is in an emotional state where it might be unsafe to drive, says Fernando Bevilacqua.
From the Group to the Individual
Today, questionnaires and physiological measurements are the most common methods used to evaluate computer games. These methods give researchers and game developers an average picture of how a group reacts to a game.
– The point of our research is to move away from the group and focus on the individual gamer's experience. It opens up for a future where you are no longer playing a game on easy, medium or hard level. Instead, you are playing the game on your own specific level, concludes Fernando Bevilacqua.
PhD thesis
Game-calibrated and user-tailored remote detection of emotions: A non-intrusive, multifactorial camera-based approach for detecting stress and boredom of players in games
By: Fernando Bevilacqua.