Sign Language Recognition from Digital Videos Using Pyramid Network with Detection Transformer

Abstract

Sign language recognition is one of the fundamental ways to assist deaf people to communicate with others. An accurate visual-based sign language recognition system using deep learning is a long-term research goal. Deep convolutional neural networks have been extensively considered in the last few years, many architectures have been proposed. Recently, Vision Transformer and other Transformers have shown apparent advantages in object recognition compared to traditional Computer Vision models such as Faster R-CNN, YOLO, SSD, and other deep learning models. In this thesis, we propose a Vision Transformer-based sign language recognition method related to DETR (Detection Transformer) to improve the current state-of-the-art sign language recognition accuracy. The method proposed in this thesis is able to recognize sign language from digital videos with high accuracy using a new Deep Learning model, ResNet152 + FPN (i.e., Feature Pyramid Network), which is based on Detection Transformer. Our experiments show that the method has excellent potential for improving sign language recognition accuracy. For instance, our newly proposed net ResNet152+FPN is able to enhance the detection accuracy by up to 1.70% on the test dataset of sign language. Besides, an overall accuracy 96.45% was achieved using the proposed method.