SPIDERS: Wearable Emotion Sensing

SPIDERS (System for Processing In-situ Bio-signal Data for Emotion Recognition and Sensing) is low-cost, lightweight and compact wearable platform that can monitor human emotions will benefit a wide area of research and applications, such as continuous health monitoring, elderly care, depression treatment, entertainment, so on and so forth. Thus, we present SPIDERS which is a low-cost, wireless, glasses-based platform for continuous in-situ monitoring of user’s facial expressions and real emotions. SPIDERS costs less than $20 to assemble and can continuously run for up to 9 hours before recharging.

Left: SPIDERS glasses platform (red: hardware sensors; blue: PCB boards; green optional attachments). Right: SPIDERS glasses on a mannequin.

Overview:

SPIDERS has a four-layer system architecutre. Each layer in the architecture directly enables the applications or functionalities in the layer directly above.

Four-layer system architecture block diagram

We present algorithms to provide four core functions (eye shape and eyebrow movements, pupillometry, zygomaticus muscle movements, and head movements). The algorithms use the bio-signals acquired from three non-contact sensors (infrared camera, proximity sensor, an inertial measuring unit). SPIDERS can be also extended to include functionalities provided by contact-based sensors, such as heart rate sensors and 8-channel EEG sensors.

Core Function Library:

We present novel and robust vision-based techniques to perform pupillometry as well as eye & eyebrow shape detection using IR-band gray-scale images, from an IR camera positioned at a low angle from the eye as to not block the field of view of the user. A proximity sensor is used to detect the movements of zygomaticus muscle (smile muscle), which aims at adding another dimension of knowledge to the facial expression.

Advanced Functionalities:

Eye images of seven common facial expressions.Theconstraints from the gray-scale image captured by a wearableeye tracker make it difficult to distinguish subtle differencesamong different emotions (eg. happy and disgusted).

SPIDERS distinguishes between different classes of facial expression and real emotion states based on these four bio-signals. We prototype advanced functionalities, including facial expression detection and real emotion classification. Specifically, we deploy a novel facial expression detector based on landmarks and optical flow that leverages changes in a user’s eyebrows and eye shapes to achieve an accuracy that outperforming the state-of-the-art approach. We also implement a pupillometry-based real emotion classifier with higher accuracy than other low-cost wearable platforms that use sensors requiring skin contact.