Development of Multimodal Agent Supporting Next-Generation “Movement”

Mar. 2, 2021

News Release

Aisin Seiki Co., Ltd. (Head Office: Kariya, Aichi Prefecture; President: Kiyotaka Ise) has developed a multimodal agent jointly with Harmonized Interactions Co., Ltd. (Head Office: Tokushima, Tokushima Prefecture; President: Norihide Kitaoka) and GarateaCircus Co., Ltd. (Head Office: Shibuya, Tokyo; President: Yuka Ishikawa) Idein Inc. (Head Office: Chiyoda, Tokyo; Representative Director: Koichi Nakamura), Toyohashi University of Technology and other organizations.

In a society where MaaS and other new forms of "movement" are required, Aisin paid attention to expectations moving ahead of people and new value to achieve this innovation through a development process that relied on diverse perspectives and application of Aisin's accumulated technologies and collaboration between academia, industry and other partners.

The developed system is intended to be used inside automatically operated buses, which are expected to become widespread in the future. It judges the situations and intentions of passengers by using camera image recognition technologies (driver monitor system and cabin monitoring system), for which Aisin has considerable expertise, to detect the movement of passengers' faces and bodies as well as multimodally analyzing detected images and voice and accumulated data by using AI.

The system can naturally communicate with passengers depending on their situation and intentions in real time in an emotionally expressive manner by utilizing the results of the judgment for the in-cabin monitoring function of preventing passengers from falling and leaving their belongings behind as well as fusing them with a CG character named Saya, created by the 3D computer graphics artist unit TELYUKA.
Saya provides fun and comfort to "movement" by identifying passengers to speak to them when they board the bus and removing the slight anxiety associated with automated driving by the monitoring function.

In the future, we aim to roll out this system not only to automatically operated buses but also to various fields for making communities and society as a whole more convenient and comfortable to live in.

* This project is supported by the Human Machine Harmonization System Consortium of the Program on Open Innovation Platform with Enterprises, Research Institutes and Academia (OPERA), Japan Science and Technology Agency (JST).