DeepMind has released a new sign-language-to-text (SL2T) model, marking a significant advancement in accessibility for Deaf and hard of hearing users. The model, which supports American Sign Language (ASL) to English, is now available in Gboard and Live Transcribe on Pixel 11 devices. This development brings sign language AI out of the lab and into consumer products for the first time, allowing users to sign to their phones in situations where they would typically type. According to the company, the feature enables Deaf users to search the web, draft messages, and interact with AI tools like Gemini through signing. Testing by users showed that signing in ASL is faster and more natural than typing in English.

The SL2T model is trained on over 100,000 hours of data across more than 50 sign languages, with a quarter of the data in ASL. This extensive training allows the model to learn shared underlying structures, outperforming single-language models in experiments. The model processes sign language inputs as sequences of pose landmark locations rather than raw video, which helps protect user privacy. It translates these coordinates directly into text, bypassing intermediate annotations known as 'glosses,' which are limited in capturing the complexity of sign languages. SL2T achieves a zero-shot score of 70 BLEURT on the FLEURS-ASL benchmark, which is significantly higher than any previously reported score.

Sign languages are the primary languages of Deaf communities and are distinct from spoken languages, with their own grammars and lexicons. They also involve simultaneous movements of the hands, arms, torso, head, and face, making them complex to translate. Early attempts at sign language technology, such as sign language gloves, were limited due to the complexity of visual perception and translation required. SL2T is designed to handle both these challenges, offering a more accurate and efficient solution. The model also addresses practical issues like minimizing streaming latency and ensuring fairness for left-handed signers.

Source: deepmind