RIT helps Google develop new technology that converts sign language into text

Gary Behm, director of the NTID Center on Access Technology, uses Google’s Sign Language-to-Text app, which he and others in the center helped evaluate prior to its release. He calls the new technology a breakthrough for users of sign language for easier communication. Researchers at Rochester Institute of Technology’s National Technical Institute for the Deaf have collaborated with Google to test new AI technology that converts sign language into text in real time. That technology, called Sign Language-to-Text (SL2T), is now available as a standard feature on Google Pixel 11 devices, released in August. It represents a significant advancement in accessible communication technology. “Sign Language-to-Text has the potential to expand communication access and empower deaf individuals around the world,” said Gary Behm, director of the NTID Center on Access Technology, which collaborated with Google on the project for the past four years by collecting data and offering feedback. “This really is a breakthrough.” Behm, who is deaf and uses American Sign Language, recalls growing up having to carry a pen and paper to help him communicate. Technology evolved over the years to enable typed text conversations on a cell phone. Now, he can turn on the phone’s camera and use sign language, which is automatically turned into text, just as someone who might voice a text message. SL2T is particularly useful and faster for individuals whose preferred mode of communication is American Sign Language. Just like a speech-to-text message, the user also has the ability to review and edit their text before it is sent. Google says it spent more than 100,000 hours training SL2T to learn more than 50 international sign languages, about a quarter of that time with American Sign Language, which is not universally used around the world and does not follow English structure. Other than reading the hand shapes, SL2T also reads facial and body movement which often define the concepts used when signing. And users only need one free hand to sign to the phone for it to convert the movement into letters, words or phrases. Behm said while the technology works well for everyday encounters, it isn’t meant to be a replacement for qualified human interpreters in medical, legal, educational, or other high-stakes situations. He credits RIT/NTID alumnus Sam Sepah for his involvement in the project, and helping improve accessibility and strengthen collaboration between the deaf community and Google, where Sepah is lead AI accessibility research program manager. “Arriving at RIT as a deaf student driven to make a difference, I built a strong foundation in leadership,” said Sepah, who earned a bachelor’s degree in multidisciplinary studies with a foundation of world history and multimedia in 2006, and his master’s degree in 2008 in Human Resources. “Through the incredible support I received there, I gained the skills that eventually led Google to select me to build their Sign Language AI Research team.” Sepah said the enduring partnership between RIT and Google, and his mentorship from Behm “is building a legacy of global accessibility technology. Together, we are showing families everywhere that with the right educational support and community, a deaf child can truly change the world.” Sepah, who founded Google DeepMind’s AI Sign Language Advisory Committee, nominated Behm to serve on it to help bring together deaf community leaders, researchers, and subject matter experts to provide guidance on the responsible development, evaluation, and deployment of the SL2T model. The committee will help ensure that new technology with AI-powered communication tools reflects the values, needs, and expectations of the deaf community. Established by the U.S. Congress in 1965, the National Technical Institute for the Deaf is the first and largest technological college in the world for deaf and hard-of-hearing students. NTID offers associate degree programs for deaf and hard-of-hearing students and provides support and access services for deaf and hard-of-hearing students who study in the other eight colleges of RIT. NTID also offers a certificate in healthcare interpretation, bachelor’s degree programs in sign language interpreting, and community development and inclusive leadership, and master’s degrees in healthcare interpretation and secondary education for individuals interested in teaching deaf and hard-of-hearing students. Deaf and hard-of-hearing students come from all over the United States and around the world to take advantage of the opportunities available to them at RIT/NTID. NASA awards RIT $2.2 million to extend Landsat satellite program collaboration RIT key partner in $15 million award to establish a national laser technology hub