Indian Sign Language Recognition System for Simple Manual Signs

Abstract

Sign language is the fundamental mode of communication among deaf community members. Each nation has its own, unique sign language. In a sign language, various hand gestures, body movements, and facial expressions are utilized to represent each sign. However, all these languages are not commonly recognized outside of these groups, there may be a communication barrier between hearing-impaired and non-hearing impaired individuals. The methods for recognizing signs created through this study enable the design of system that can help to reduce this barrier, either by giving computer tools to aid in the acquisition of sign language or, possibly, by creating portable sign-to-speech translation systems. The research work presents the detailed description about the general process of sign language recognition system using two different datasets (static and dynamic) of signs. A systematic literature review related to the Sign Language Recognition System (SLRS) for static and dynamic signs is depicted in this research work. The current status of sign language recognition system w.r.t the dataset is classified into static and dynamic signs. On the basis of published works, the periodic development of sign language recognition and research studies has been evaluated. In addition, the review methodology is followed and provided, and sources of publications and research papers are retrieved according to inclusion-exclusion criteria. This study methodology will aid in the dissemination of results in a methodical manner, therefore allowing researchers working in comparable fields to pick the most effective strategies for recognizing static and dynamic signs of Indian sign language (ISL). As no public dataset is available for the recognition of Indian signs this thesis presents the collection and development of datasets for static signs as well as for dynamic signs. It also describes the detailed procedure about how the dataset has been collected from the number of users under different environmental conditions and at different distances.xix This thesis also presents different architectures for sign language recognition of static and dynamic signs of ISL. In this MediaPipe Hand and MediaPipe Pose techniques are used as data pre-processing for efficiently recognize static and dynamic signs. The SLRS described in this thesis is developed using deep learning based techniques. In this, different convolutional neural network architectures have been compared and the results are analyzed on the basis of accuracy, precision, recall, F1-score and loss curves. The implemented convolutional neural network architecture not only helps to enhance the accuracy of the model but also helps to increase the efficiency of the model. The experimental analysis show that the implemented model outperforms the traditional machine learning algorithms for sign language recognition. Further, to recognize Indian signs at real-time different Convolutional Neural Network (CNN) architectures like Visual Geometry Group 16 (VGG16), VGG19 and GoogleNet are implemented and compared. It has been observed from the experimental analysis that VGG19 architecture using MediaPipe technique and CNN architecture using MediaPipe outperformed all the other CNN based architectures for static sign recognition and dynamic sign recognition respectively. This thesis also presents the developed Progressive Web Application of the proposed system. This web application promotes the communication between hearing-impaired and non-hearing impaired people. It serves the purpose to expedite the users to recognize different static signs in real-time. The goal to develop such an application is to outreach the hearing-impaired people to communicate with other persons in the society and learn new facts. This system can also help hearing-impaired people to get education, enhance their skills and make their career