Share Email Print

Proceedings Paper

Fusion strategies for speech and handwriting modalities in HCI
Author(s): Claus Vielhauer; Sascha Schimke; Valsamakis Thanassis; Yannis Stylianou
Format Member Price Non-Member Price
PDF $17.00 $21.00

Paper Abstract

In this paper we present a strategy for handling of multimodal signals from pen-based mobile devices for Human to Computer Interaction (HCI), where our focus is on the modalities of spoken and handwritten inputs. Each modality for itself is quite well understood, as the exhaustive literature demonstrates, although still a number of challenges exist, like recognition result improvements. Among the potentials in multimodal HCI are improvements in recognition and robustness as well as seamless men-machine communication based on fusion of different modalities by exploiting redundancies among these modalities. However, such valuable fusion of both modalities still poses some problems. Open problems today include design approaches for fusion strategies and with the increasing number of mobile and pen-based computers, particularly techniques for fusion of handwriting and speech appear to have a great potential. But today few publications can be found that addresses this potential. In this work we introduce a conceptional approach based on a model to describe a bimodal HCI process. We analyze four exemplary applications with respect to the structure of this model, and highlight the open problems within these applications. Further, we will outline possible solutions to these challenges. Having such fusion model for HCI may simplify the development of seamless and intuitive to user interfaces on pen-based mobile devices. For one of our application scenarios, a bimodal system for form data recording and recognition in medical or financial environment, we will present some first experimental results.

Paper Details

Date Published: 14 March 2005
PDF: 9 pages
Proc. SPIE 5684, Multimedia on Mobile Devices, (14 March 2005); doi: 10.1117/12.585869
Show Author Affiliations
Claus Vielhauer, Otto-von-Guericke-Univ. Magdeburg (Germany)
Sascha Schimke, Otto-von-Guericke-Univ. Magdeburg (Germany)
Valsamakis Thanassis, Univ. of Crete (Greece)
Yannis Stylianou, Univ. of Crete (Greece)

Published in SPIE Proceedings Vol. 5684:
Multimedia on Mobile Devices
Reiner Creutzburg; Jarmo H. Takala, Editor(s)

© SPIE. Terms of Use
Back to Top
Sign in to read the full article
Create a free SPIE account to get access to
premium articles and original research
Forgot your username?