A perceptive remote-control 3D TV interface

3D TV is a turning point in the history of television. Here, technologies from computer graphics, computer vision, multimedia, telecommunications, broadcasting, and other related fields merge to expand the sensations provided by watching conventional 2D TV.¹ As a result, there has been much research on multi-view image processing and 3D display.^2,3 However, most of the previous studies have focused on acquiring and displaying images while optimizing their 'naturalness', simplicity of computation, and real-time operation. Free-viewpoint TV (FTV)⁴ takes this idea forward. Free-viewpoint systems allow a viewer to look at a scene from the perspective that they want rather than what a director chooses to provide.¹ In order to generate a scene according to the user's requirements, a position in 3D space has to be defined as the view point.

To achieve this, we suggest an intelligent approach that automatically generates reference viewpoints based on a theory of human visual attention. Many studies of visual attention and eye movements have shown that humans generally only attend to a few areas in an image rather than scanning the whole, and visual attention models provide a general approach to controlling the activities of active vision systems.⁵ These models of selective visual attention have been suggested on the basis of evidence in psychology, psychophysics, physiology, etc., and can also be exploited for our purposes.

Figure 1 shows an examples of how our artificial human visual attention system can be used to obtain a basis position to generate a scene that corresponds to a user's viewpoint. The system also simultaneously computes the intensity of salience of an object in a given region. In fact, several objects or regions can be detected in terms of their their individual saliencies, which can in turn be used to describe them. The system computes early visual features from a set of pre-attentive feature maps in a massively parallel way. Activity from all feature maps is combined at each location, giving rise to the responses in the topographic saliency map shown in Figure 2.

In our perceptive remote-control system, the focuses of attention-as determined by our model-are displayed and candidate viewpoints presented (see Figure 3). The user then chooses from them using a touch screen. The proposed remote controller system is smart in that it provides the viewers with several interesting viewpoints without disorienting them at the same time.

In our experiment, the 3D TV is realized using a networked desktop computer (see Figure 4). The 3D TV is networked with the PDA (smart remote controller) using a wireless link. Zigbee and ultra-wideband (UWB) are possible candidates for this networking in future.

Free-viewpoint television, despite being a great advance, could initially prove difficult for some people, requiring users to acquire a level of skill before they can enjoy it fully. Our proposed remote controller works out this problem by providing the user with reference viewpoints, second-guessing their likely interest based on human visual attention models.We believe that this is one of the first time that user's (rather than producer's) perspective on the technology has been properly addressed, and that it is therefore an important contribution to the technology.