BRAIN ROBOT AUGMENTED INTERACTION

CHUNGHO LEE A, JINUNG AN 

EXPERT SYSTEMS WITH APPLICATIONS

​

LSTM-CNN MODEL OF DROWSINESS DETECTION FROM MULTIPLE CONSCIOUSNESS STATES ACQUIRED BY EEG

This study aimed to design a deep neural network for electroencephalography (EEG)-based drowsiness detection in multiple consciousness states, i.e., “awake,” “sleep,” and “drowsiness.” Few studies have seriously considered the optimal input vector size or labeling method in classifying multiple consciousness states, which may affect classification performance. To determine the optimal input vector length, i.e., window length, three neural network models (long short-term memory [LSTM], convolutional neural network [CNN], and combined LSTM and CNN) and four feature-based models were tested with six different levels of window length. The EEG dataset was acquired from 19 participants with randomly assigned auditory stimuli and button responses. The EEG data were labeled into three classes (awake, sleep, and drowsiness) based on the defined button response pattern corresponding to the stimuli. The results demonstrated that when the input vector size exceeded 8 sec, the performance of the neural network models dropped rapidly; however, when the window size was less than 8 sec, the performance change according to the window size was small. In contrast, the performance of feature-based models increased continuously as the window size increased. The LSTM model yielded the best accuracy (86%) for a 1 sec window length, and the LSTM-CNN model yielded the best kappa index (0.77) for a 4 sec window length. In addition, the proposed model was applied to the binary classification of normal consciousness (awake) and low consciousness (drowsiness and sleep) states to determine whether this model works appropriately in actual applications such as drowsiness detection in a driving environment. For binary classification, the LSTM-CNN model resulted in 0.95 F1 scores in 4000-ms. When a short input data (500 msec) is used, the LSTM-CNN model resulted in an average accuracy of 85.6% and a kappa index of 0.77 for the three-class classification problem and 0.94 F1 scores for the binary classification problem. In conclusion, we demonstrated that the proposed model could effectively detect drowsiness. Furthermore, a significant correlation was found between reaction time and drowsiness. However, using the reaction time as an index for labeling drowsiness was challenging because of the high false-negative ratio.