Real time sign language translator for video conferencing platforms

Abstract

Sign Language is the method of communication of mute people all over the world. There are about 70 Million sign language users around the world. But only a few percent of people who can hear and speak know sign language. This makes it difficult for mute people to communicate. Computer-based Sign Language Recognition is a breakthrough technology to overcome this problem. After pandemic businesses and organizations have started adapting online video conferencing platforms for carrying out meetings, workshops, interviews, collaborations, etc. The aim of this project is to provide a practical solution for sign language interpretation. Here we propose a lightweight real-time and integrable sign language detection application, that can be used in any video conferencing platform such as google meet, microsoft teams, zoom, discord, etc. Here we have used deep learning algorithms, image processing and the concept of virtual cameras to achieve our goal. We describe a desktop application to sign language detection in the browser in order to demonstrate its usage possibility in videoconferencing applications. We use the MediaPipe Holistic pipeline and LSTM for pose detection and to train and predict sign languages. It shows 91%-93% prediction accuracy while the latency is still under 4ms. Keywords: Sign Language Translation,American Sign Language, LSTM, Virtual Camera, Hand Gesture Recognition, OpenCV, video-conferencing, Sign Language Translator for Meeting Apps, Real Time Sign Language Translation, Google Meet, Microsoft Teams, Zoom.