Artificial Intelligence (A.I.) for ASL learning?

We can’t emphasize enough how important it is for us to keep the Deaf community’s best interest in mind. Our ASL expert gave an interview to Deaf activist, journalist, and ASL teacher Melissa Elmira Yingst (a.k.a "Melmira") about our A.I.-based solution for learning sign language, SignAll Lab. In the interview, they openly address the uneasiness that many ASL teachers have when they learn about this new technology – will A.I. steal their jobs? Watch the video to get the answer! 

Video Transcript

[Description: On an all white background, black text appears: “1. Learn”, then “2. Practice” and “3. Quiz” with SignAll’s logo. On-screen text reads “SignAll, new technology for ASL”. A screen recording of the app, and a person with signing gloves to use as AI technology for fingerspelling practice. Melmira appears on screen wearing a gray top and funky earrings.]

Melissa: Hola! For a long time, sign language has been popular for many, not just our community but many hearing people who want to learn. As an ASL teacher myself, of course when I think of learning sign language, it’s important to have a person - as in a real human - to teach, right? At the same time, technology has been rapidly advancing lately and we have to keep up with technology. So, sign language with technology? Interesting. I’m intrigued by SignAll’s program that uses AI to teach sign language. What? Check it out.

[Recording of a person teaching the sign for “college”. Then there is a video of another man sitting wearing gloves. The man replicates the words and signs the teacher is spelling, and a gold star appears if the signer wearing gloves replicates the letters and signs correctly. Video clip of an audience watching this demonstration on a screen. Melmira’s logo appears in front of the video. Supported by Convo and DPAN.TV. A man appears on screen. The SignAll logo appears on the left side as the man signs. On-screen text reads “Jesada Pua, SignAll ASL expert”.]

Jesada: This company has two different main products. Our original product is called the automatic sign language recognition translation system. It’s Deaf-made. That means gloves with different colors on each finger so that our computer system, our platform, can recognize hand movements and translate it to written slash spoken English. That has been an ongoing project and we realized it has some political and ethical issues. Plus technical issues, of course! So we realized this technology needs more community input which we’re working on. That led us to launching a main product called virtual ASL learning. It’s really cool. A quick summary. You know, today there are many hot fads with technology, right? I’m sure some of you have already played a game called Wii with those white gadgets or tools. For example, if you play a game like tennis, you would use those gadgets and their platform can recognize your movements as if you’re playing a virtual game. That’s the same concept as what we’re using for our system. New students, who are hearing most of the time, can use gloves to sign! If they sign wrong, the system will identify that they’re wrong.

Melissa: Cool! I like that Jesada recognizes this needs community input. Now, I’m curious to see this in action!

[A light comes on. Roll over to a screen. “Just a moment… Loading”. A person wearing a glove with different colors on the fingers spells “U”, “Q”, and “V”. Video on the screen showing technology identifying her hand shapes and movements and identifying the letters. With each letter, a gold star appears. The person makes an unidentifiable hand shape and a question mark appears. Then she spells out “O” and a gold star appears. Close up of a SignAll page on a computer. Close up of a camera then zoom in to the same person fingerspelling with the glove on. Video of this individual on the left hand side of the screen with a teacher on the left spelling out the same letters. The top of the screen reads “Fingerspelling Letters Q-Z”. The video continues as Jesada appears on the right side.]

Jesada: So, how does this benefit those who are learning sign language? There are many benefits! I’ll share some. For example, imagine students who learn ASL in a traditional classroom. They can’t just do “good enough”. They must practice, which often happens after school, right? So they have Deaf events to go to to practice, but it’s true that Deaf people often value their Deaf space. Our system seems to be one great solution to that. For example, if a hearing person learns sign from a book, video such as through YouTube, or online, they tend to perceive and express it incorrectly. For example, how you sign “year” is in this movement but it’s common for learners to make a different motion. Who will tell them whether they’re doing it wrong or right? Our system has a camera that will detect any wrong movements and notify them of the correct way to do it. That’s a cool benefit, as our system is more interactive than learning through books, or the internet. Those resources are 2D, while our system is 3D but actually more 4D because it includes time and speed which will give students a better feeling and understanding of how to sign correctly. Another benefit is that it will help learners remember signs. In a classroom, after class that’s it. Where is the training after school? Our system is interactive and uses gloves to train them and help them review. This truly accelerates their memory. Also, some students have social anxiety which means learning in classes and then meeting people in the Deaf community might be difficult or too much. Because this system is designed for a person to use it alone, that means you can practice alone and gain enough confidence to go out and socialize with the community.

Melissa: This is definitely something that ASL students can use to add to their existing learning from a classroom taught by a Deaf ASL instructor. You know, ownership is really important with Deaf ASL teachers. We’re already having a hot discussion about the question of whether hearing people should even teach ASL. So with this program, of course I thought about the possibility of this taking away opportunities from Deaf ASL teachers.

Jesada: Yes. This is definitely a hot topic. Will this replace ASL teachers and take their jobs? Simple answer: No. For many years, since the first day of documented history, humans always have used language as the most powerful communication tool to communicate with each other. So we will always need actual teachers to teach. For example, if a person learns Spanish in a traditional classroom, that’s the ideal way to learn. Learning from a book isn’t enough. You need to learn from a person because humans are social animals, right? We need language and connection. However, where can people practice after the classroom? Hearing people tend to use - I’m sure some of you already heard about that app called Duolingo. That’s an awesome app that helps people learn English and like twenty different languages. It’s a cool app. So why not add ASL to that? After you learn ASL from a classroom, how can you continue to remember the ASL you’ve learned? Duolingo and our system is more like an amendment and an after school activity. It’s almost like homework. It’s one way to look at it. We will always need actual human teachers to teach any language, really. We’re more like an interactive, after-class program. And we can say we already have up to ten different pilot systems established in the U.S. in a range of places from high schools to community colleges to universities. And reviews have been positive. Teachers and students have given positive reviews. They say it helps them significantly improve their signing.

Melissa: While we’re touching on the topic of AI, I have to ask. Why is that something we should learn about and embrace rather than perceive as “too much” and push aside?

[Jesada appears again on the left side of the screen. A screen recording of the program plays - it is titled “Spaceman”, a replacement for Hangman, and a man on the left tries to guess different letters and receives a gold star every time he guesses and spells a letter correctly.]

Jesada: Actually, I get that question often. Why should we not be afraid of AI? A lot of people think of AI as something out of a Hollywood movie like robots taking over the world and controlling us. There are different levels. For example, how did Deaf people order food before? We had to call to have food delivered. Now we can just use apps to have our food delivered. AI always has a certain line that continues to shift. Here’s a situation as an example. I traveled in East Europe and didn’t understand a country’s language. There was a situation where I tried to communicate with a local person and we weren’t able to get our messages across. So I used Google Translate, which can translate English to over 50 languages. That way, we finally got what we wanted. Why not add ASL to Google Translate? Even though most Deaf people can write and read, linguists view ASL as a language. So why not give them an equal linguistics status like any spoken language? There are some certain situations where AI does make our lives easier. You don’t have to go to the bank - you can just use an app. There are many different reasons. Our system products are not just for learning. We have many games like Hangman, crosswords, and many others. It’s really champ because even though ASL is a cherished cultural aspect of the Deaf community, it can be used with AI as well. Plus, with coronavirus, many schools are still closed for the fall. Some are open, but with class limits like five to ten students. How can those students practice signing? Our system means students can practice their signing.

[Rose petals fill the screen.]

Melissa: Thank you, Jesada! This really made me think about different topics - AI for learning sign language and how we Deaf people can really keep up with technology. What are your thoughts?

[Melmira’s logo appears on screen. Supported by Convo and DPAN.TV. Host/Anchor: Melissa Elmira Yingst. Edited by Alexandra Hickox. Captioned by Brianna Keogh.]