CoeFont Provides Free Service to Osaka University’s Lip2ja, a Lip-Reading-Based Japanese Speech System

Tokyo, Japan — [Date] — CoeFont Inc., a provider of AI voice platform “CoeFont,” is pleased to announce its free service provision to “Lip2ja,” a Japanese speech system developed by a research group at Osaka University Graduate School of Information Science and Engineering, led by Associate Professor Yoshihiro Godo, Professor Norikazu Miura, and Professor Hidenori Igarashi at the Graduate School of Medicine. This collaboration aims to improve the quality of life (QOL) of individuals who have lost their voices due to illness or other conditions by enabling them to communicate through their lip movements.

Overview of the Initiative

CoeFont is providing its high-quality AI voice technology to the Lip2ja system, which uses AI to estimate spoken words from video footage of a person’s mouth captured by a smartphone camera. The estimated text is then input into CoeFont to generate actual speech. By registering a person’s voice before surgery, Lip2ja can reproduce the lost voice based solely on lip movements after surgery, contributing to the improved QOL of both the patient and their family members.

Device Overview

Developed at Osaka University, the AI analyzes video footage of a person’s mouth captured by a smartphone camera to estimate the spoken words. This estimated text is then input into CoeFont to generate actual speech. By pre-registering a person’s voice before surgery, CoeFont can reproduce the lost voice after surgery based solely on lip movements, significantly improving the QOL of both the patient and their family members. A presentation on this device will be given at the 75th Annual Meeting of the Japan Society of Tracheobronchology on October 15, 2024.

Comment by Yoshihiro Godo, Specially Appointed Associate Professor, Graduate School of Information Science and Engineering, Osaka University

“In this joint research project between Osaka University’s medical and engineering departments, we aim to create a barrier-free communication environment through speech. We are developing a system that provides communication support as close as possible to the current speaking state, even for those who unfortunately have difficulty speaking naturally.

I first encountered CoeFont at the 125th Annual Meeting of the Japan Society of Otolaryngology-Head and Neck Surgery in May 2024. I was amazed by the high quality of the AI voice. Thanks to their swift support for our research, we have been able to significantly improve our system, which enables people to speak in their own voices through lip reading.

When we recall old memories with family and friends, it’s not uncommon for vivid voices to be awakened along with the images. I hope that our developed system will not only improve the quality of daily life but also add a colorful voice to precious memories.”

Yoshihiro Mido Profile

• 2006: Ph.D. (Information Science), Osaka University
• 2006: Specially Appointed Researcher, Graduate School of Information Science and Technology, Osaka University
• 2007: Assistant Professor, Graduate School of Information Science and Technology, Osaka University
• 2021: Specially Appointed Associate Professor, Graduate School of Information Science and Technology, Osaka University