Share Email Print

Proceedings Paper

Integrated multimodal human-computer interface and augmented reality for interactive display applications
Author(s): Marius S. Vassiliou; Venkataraman Sundareswaran; S. Chen; Reinhold Behringer; Clement K. Tam; M. Chan; Phil T. Bangayan; Joshua H. McGee
Format Member Price Non-Member Price
PDF $17.00 $21.00
cover GOOD NEWS! Your organization subscribes to the SPIE Digital Library. You may be able to download this paper for free. Check Access

Paper Abstract

We describe new systems for improved integrated multimodal human-computer interaction and augmented reality for a diverse array of applications, including future advanced cockpits, tactical operations centers, and others. We have developed an integrated display system featuring: speech recognition of multiple concurrent users equipped with both standard air- coupled microphones and novel throat-coupled sensors (developed at Army Research Labs for increased noise immunity); lip reading for improving speech recognition accuracy in noisy environments, three-dimensional spatialized audio for improved display of warnings, alerts, and other information; wireless, coordinated handheld-PC control of a large display; real-time display of data and inferences from wireless integrated networked sensors with on-board signal processing and discrimination; gesture control with disambiguated point-and-speak capability; head- and eye- tracking coupled with speech recognition for 'look-and-speak' interaction; and integrated tetherless augmented reality on a wearable computer. The various interaction modalities (speech recognition, 3D audio, eyetracking, etc.) are implemented a 'modality servers' in an Internet-based client-server architecture. Each modality server encapsulates and exposes commercial and research software packages, presenting a socket network interface that is abstracted to a high-level interface, minimizing both vendor dependencies and required changes on the client side as the server's technology improves.

Paper Details

Date Published: 28 August 2000
PDF: 10 pages
Proc. SPIE 4022, Cockpit Displays VII: Displays for Defense Applications, (28 August 2000); doi: 10.1117/12.397779
Show Author Affiliations
Marius S. Vassiliou, Rockwell Science Ctr. (United States)
Venkataraman Sundareswaran, Rockwell Science Ctr. (United States)
S. Chen, Rockwell Science Ctr. (United States)
Reinhold Behringer, Rockwell Science Ctr. (United States)
Clement K. Tam, Rockwell Science Ctr. (United States)
M. Chan, Rockwell Science Ctr. (United States)
Phil T. Bangayan, Rockwell Science Ctr. (United States)
Joshua H. McGee, Rockwell Science Ctr. (United States)

Published in SPIE Proceedings Vol. 4022:
Cockpit Displays VII: Displays for Defense Applications
Darrel G. Hopper, Editor(s)

© SPIE. Terms of Use
Back to Top