Multilingual User Gaming Interaction: let’s talk about D-Box

15 January 2015

Imagine a game between several users, speaking different languages and interfacing using a same voice.

Imagine a game between several users, speaking different languages. And imagine them playing without caring about language barriers, because the interface is taking care of that. Then imagine that they all hear the on-going conversation with the same voice, but each in his or her own language. Because the same voice speaks several languages. We are talking about Multilingual User Gaming Interaction, the aim of the D-Box project.

D-Box’s main goal is to develop and test an innovative architecture for conversational agents whose purpose is to support multilingual collaboration between users on a common problem in an interactive application.

The interactive agent will enable type-written and/or spoken collaboration in the users’ native language by mediating communication: all user interactions will be transmitted through the D-Box multilingual agent which will (1) manage and support dialogic interactions based on its understanding of the users’ shared goals, and (2) process, extract, and transfer knowledge among users. The agent will have no translation capabilities, but will instead have access to rich linguistic resources in multiple languages and domain-and application-specific knowledge bases.

The project is funded by EUREKA, the Pan-European research and development funding and coordination organization. The project leader is Mi’pu’mi Games GmbH and the project partners are Acapela Group, Idiap Research Institute, Koemei SA, Sikom Software GmbH and Universität des Saarlandes.

‘Gaming has not yet entered the speech dimension and the D-Box project is a great opportunity to introduce real conversational capabilities to new games in the very near future. We are looking forward to seeing and hearing developers design their games with conversation at the heart of their game architecture. We are very enthusiastic about being part of this project, about contributing our expertise and about working with such prestigious partners’ comments Lars-Erik Larsson, CEO of Acapela Group.

From a research point of view, the project requires building a multilingual conversational agent which will seamlessly interact with multiple users speaking different languages and driven by a common goal defined by the game. This involves the development and integration of multilingual speech recognition systems, multilingual speech synthesis, multilingual dialogue modelling, and cross-domain adaptation resources.

From an integration and evaluation point of view the project’s key innovative idea is that the overall anticipated framework will be application-agnostic.

D-Box is concerned with addressing scalability, a common problem with today’s interactive conversational systems, through the development of a conversational agent that is cheaply and easily portable across languages and adaptable to different domains and environments. In order to ensure portability, the framework developed in the context of a game scenario will be in parallel evaluated and tested in the context of a standard commercial voice-based dialogue system.

The system consists of several modules: Dialogue Engine (DE), Automatic Speech Recognition (ASR), Text-to-Speech synthesizer (TTS), Communication module (COM-Modul), Audio Manager, Game-Client and Game-Server, all connected and communicating with each other.

The D-Box project will use three EU languages: English, German and French.

The goals of D-Box are challenging and full of application potential. Research is conducted by a consortium of leading edge experts in the different fields, strongly committed to the project.