Investigations on Sleep Deprivation Induced Fatigue Detection Using Multimodal Fusion

Abstract

Sleep is a fundamental biological need that is often overlooked in today’s fast-paced and busy world. In our pursuit of productivity and success, many individuals sacrifice sleep, unaware of the profound consequences. A sleep-deprived fatigued person is prone to commit mistakes that could lead towards fatality. In today’s era, sleep deprivation has become a pervasive problem that affects people of all ages and walks of life. With increasing responsibilities, technological distractions, and lifestyle choices, many individuals fail to prioritize sleep, leading to a chronic lack of rest. Youngsters spending hours on phone screens before going to bed is also another reason for prevailing sleep deprivation. Sleep deprivation concerns the insufficient sleep needed due to which an individual does not feel fully rested and alert during the daytime. While the recommended duration varies with age, for optimal functioning, adults need 7-9 hours of sleep per night. However, surveys indicate that a significant portion of the population, especially in urban settings, regularly experiences sleep deprivation. Factors contributing to this epidemic include long working hours, excessive screen time, social obligations, and untreated sleep disorders. Therefore, it is essential to assess the fatigue. Furthermore, sleep-deprived individuals experience slower reaction times, reduced alertness, and impaired decision-making skills. These cognitive deficits can be particularly dangerous in critical situations, such as when operating heavy machinery or driving, leading to an increased risk of accidents and injuries. Sleep deprivation undermines performance and productivity across all aspects of life. In the workplace, it can lead to reduced efficiency, decreased creativity, and an increased number of errors. The employees who are Sleep-deprived are likely to struggle with decision-making and focusing, hindering their ability to perform at their best. While reviewing the available literature we have identified that the most of existing methods to detect fatigue are uni-modal and intrusive and the major demerit of these methods is that an individual remains aware of being monitored while obtaining the samples, which results in biases. The proposed methodology is a novel approach in terms of detecting the fatigue non-intrusively and through multimodal feature fusion. In our approach, fatigue is detected by obtaining significant features from four domains: visual facial images, thermal images, voice analysis, and keystroke analysis. In the proposed approach, the samples were acquired from the volunteers; considered as subjects for feature extraction, and empirical weights were assigned to the four different domains. Healthy and young individuals between the age group of 20 to 30 years age group participated voluntarily in the experimental study. Additionally, they abstained from consuming alcohol, tea, and coffee; or medication affecting the pattern of sleep. Initially, the model was trained with 80 samples (40 from Alert subjects and 40 from Fatigued subjects). Afterward, we tested it on the remaining 40 samples (20 from Alert subjects and 20 from fatigued subjects) and finally cross-validation was performed to evaluate the baseline performance. The results are finally compared with five different classifiers: k-nearest neighbors (kNN), random tree, support vector machines (SVM), random forest, and multilayer perceptron classifiers. The proposed multimodal feature fusion methodology has achieved an average accuracy of 93.33% in 3-fold cross-validation to detect fatigue non-intrusively. To enhance the accuracy, we implemented the adaptive learning method and performed trials on 60 subjects in two states (Alert and Fatigued). We obtained a total of 120 samples, 60 samples for the alert state and 60 samples for the fatigued state. Initially, we experimented it only for one domain i.e., through eye cues. In this method, the 'image profile' function was used upon the segmented images of eyes to categorize the open-eye frames and closed-eye frames. Additionally, we have identified a new feature utilizing the inter-frame interval of a closed and open eye to differentiate the Alert and Fatigued state through an adaptive threshold. The proposed approach is self-learning that works in real-time, and it achieved a detection accuracy of 97.5 %. Its accuracy and sensitivity were likely to be increased further by increasing the size of the data set because it is self-learning and real-time in nature. Subsequently, we tried the adaptive learning methodology on multimodal features and increased the sample size to 180 samples, 80 samples for the alert state and 80 samples for the fatigued state. We have proposed a self-learning framework for multimodal feature fusion from four different domains (visual spectra images, thermal images, voice, and keystrokes). We trained our model with 120 samples (60 for the Alert state and 60 for Fatigued state) and then tested it on the remaining 60 samples (30 from the Alert state and 30 for Fatigued state). The results are compared with k-nearest neighbors (kNN), support vector machines (SVM), random tree, random forest, and multilayer perceptron classifiers for cross-validation. The highest accuracy of 98.33% was achieved to detect fatigue non-intrusively. This experiment shows that the performance is improved after implementing the adaptive self-learning technique on the multimodal feature fusion.