Decoding Cognitive States from fNIRS Neuroimaging Data Using Machine Learning

Liu, Ruixue

Etd

Decoding Cognitive States from fNIRS Neuroimaging Data Using Machine Learning

Public

Building brain-computer interfaces that can automatically adapt to an individual's changing cognitive states has important implications in many domains, such as gaming, driving, and learning. Recently, the use of functional near-infrared spectroscopy (fNIRS) has received focus because of its promise for detecting an individual's cognitive state in more ecologically valid studies. In this dissertation, we focus on improving and expanding the usability of fNIRS for brain-computer interaction research. Particularly, we investigated the feasibility of using fNIRS to identify several user states that occur frequently in human-computer interaction, and that could inform adaptive user interfaces, but that are difficult to detect. We accomplished this goal by designing and conducting three human subjects experiments, collecting and curating fNIRS datasets, as well as developing and applying novel machine learning methods appropriate for the particular classification problem and that are tuned to the characteristics of fNIRS data. Particularly, we: 1. Explore mind wandering detection using fNIRS and develop a machine learning framework to incorporate individuals' differences in hemodynamic responses. Specifically, we conducted a study using fNIRS during the Sustained Attention to Response Task (SART) task to elicit mind-wandering states. We then built machine learning classifiers both on an individual level and at a group level to classify mind-wandering state versus on-task state. We also propose an individual-based novel window selection algorithm to incorporate individuals' differences in time window selection. Our results show that the proposed algorithm achieves significant improvements over the previous state-of-the-art in terms of brain-based detection of mind-wandering. 2. Explore driver cognitive load classification using fNIRS and investigate machine learning techniques for extracting spatial and temporal patterns from fNIRS data. Specifically, we conducted a study using fNIRS in a driving simulator with the n-back task used as a secondary task to impart structured cognitive load on drivers. We apply Convolutional Neural Networks (CNNs), multivariate Long Short Term Memory Fully Convolutional Networks (LSTM-FCNs), and Echo State Networks (ESNs) for fNIRS feature extraction and classification. Our results show that ESNs achieve state-of-the-art classification results for classifying different levels of driver cognitive load. 3. Explore cognitive processes associated with positive and negative learning outcomes using fNIRS and validate the generalizability of the proposed ESN models across tasks. Specifically, we conducted another study using fNIRS during a rule-learning task. We compare the classification results of CNNs, LSTM-FCNs, and ESNs for differentiating successful and unsuccessful rule learning processes. Our results show that ESNs achieve superior classification results and can extract distinct temporal patterns for different cognitive processes based on fNIRS data. By improving and expanding the usability of fNIRS for identifying important user states for human-computer interaction, the results from this research serve as a foundation for future work that integrates fNIRS data for measuring an individual's changing cognitive states. Furthermore, findings from this work have important implications for building fNIRS-based brain-computer interfaces that can automatically adapt their behavior to better support the user and provide a better user experience.

Creator