Gesture-to-Text and Speech System

To improve an earlier gesture-to-text and speech system into a more complete, usable, and well-tested prototype.

The previous system had one-way communication, slow text formation, and limited interaction. This project adds dual-hand detection, auto-suggestion, speech-to-text, and more.

Assisting users with difficulties in speaking, hearing, or interacting through conventional input methods.

In this project, we worked on improving our earlier gesture-to-text and speech system. The previous version was able to detect hand gestures and convert them into text and speech, but it had several limitations such as one-way communication, slow text formation, and limited interaction features. In the current phase, we extended the system by adding dual-hand detection, auto-suggestion, next- word prediction, speech-to-text support, air whiteboard functionality, dynamic gesture recognition, and abrowser-basedinterface. Media Pipewasusedforhandlandmarkdetection, whilelightweightlogicwas used for suggestions and motion-based gesture detection so that the system could still run on a normal laptop. We also focused on improving the project from a technical and evaluation point of view. Hardware testing, software testing, and performance observations were included to better understand where the system performs well and where it remains weak. During testing, static alphabet gestures were found to bemorereliablethandynamicgestures, whiledynamicgestureswerestillusefulforcontrol-typeactions. The final system is more interactive, more complete, and better documented than the earlier version.

1

=== Page 3 === Contents

Abstract 1