Advancing ultra-realistic communication in Japan

In the future, communication will very likely evolve from what we know as telecommunication, i.e., simply transmitting information, to new forms of interaction that are even more valuable because they combine human mutual understanding and cooperation through a network. In particular, ultra-realistic communication—which enables people who are far apart to feel as though they are in the same room—will be extremely important in ensuring, for example, that the elderly live in convenience and comfort in an information society.

Because ultra-realistic communication involves many different technologies, a wide range of specialists—such as video and audio engineers, content creators, psychologists, and cognitive scientists—will be essential to successful realization of the effort. Consequently, government, industry, and academic institutions need to share their knowledge and strive toward standardization. At the National Institute of Information and Communications Technology (NICT) in Japan, we have established the Ultra-Realistic Communications Forum (URCF) with the objective to bring together researchers, businesses, and users from many fields to exchange relevant information and ideas and to promote effective research and development, implementation trials, and standard protocols. The institute has 102 corporate members and 101 special members.¹

Figure 1. Image formats of high-definition TV (HDTV) and Super Hi-Vision.

Here, we review Japanese initiatives in ultra-realistic communication, especially regarding 3DTV and ultra-high-definition TV (HDTV).² In December 2007, Nippon BS Broadcasting Corporation began pioneering stereo 3DTV service via satellite. In this implementation, left and right images from a binocular stereo camera are compressed horizontally and composed into one HDTV video signal that can be transmitted by a normal HDTV digital satellite-broadcasting channel. At a 3DTV receiver, the signal is separated and expanded to regenerate the binocular images. Panasonic developed a stereo 3DTV system with HDTV resolution that consists of a Blu-ray disc recorder and a plasma display running at 120 frames per second. The Beijing Olympic Games provided a convincing demonstration of this system.

Super Hi-Vision, being developed by the Japan Broadcasting Corporation (NHK), is ultra-high-definition video that has four times the pixels of HDTV both horizontally and vertically. The texture is so fine that, even near the screen, an observer cannot make out the pixels. As a result, images can be viewed as close as 0.75 times the height of the screen. At that proximity, the angle of viewing is 100 degrees, which confers an enhanced sense of reality (see Figure 1). NHK is conducting research on compact cameras, H.264 video encoders, digital satellite-transmission systems, and displays for the home, with the aim of starting broadcasting service beginning in 2025.

Figure 2. Experimental setup for electronic holography.

NICT is the only public body in Japan to specialize in information and communication.³ The institute's efforts are focused on mutisensory aspects—e.g., touch, hearing, and smell—of ultra-realistic communication. We are taking two different approaches to this problem: researching and clarifying human perceptual and cognitive mechanisms in establishing realistic ambience and immersive environments, and constructing and validating prototypes.

Few of the general mechanisms that connect the senses are well understood, in particular sight/hearing, sight/touch, and sight/smell, or even sight/hearing/smell. We aim to clarify the means by which multisensory information is combined by analyzing psychophysical experiments, neurological measurements, and biological signals such as pulse, breathing, perspiration, and pupil diameter. For example, using functional magnetic resonance imaging, we are investigating how humans sense ‘presence’ and how the brain is activated when watching 3DTV.

In our work on prototypes, we are researching electronic holography that accurately reproduces the same light reflected or emitted from objects in another place (see Figure 2). Holographic technology was invented more than 50 years ago and has been applied in still images such as photographs. However, our objective is to apply the technology to moving images to transmit live pictures in real time. To create holographic movies, an electronic device such as an LCD is used instead of a photograph. But the resolution of current devices is insufficient, and no satisfactory visible range has been produced. We would like to expand the visible range by taking a systematic approach that uses several layers of LCDs. For example, we enlarged the horizontal viewing zone by combining each reconstructed beam from three LCD panels. As a next step, we intend to position each LCD panel very precisely and, moreover, to control beam intensity so that the reconstructed images in each of the divided viewing areas will be uniformly bright. In addition, we will be looking into how to obtain holographic images from objects under natural light without using laser optics.

In June 2008, Japan's Cabinet Office and Council for Science and Technology Policy selected 23 innovative technologies for which the country should proactively pursue research and development. 3D imaging was one of them. Furthermore, in June 2008, the Ministry of Internal Affairs and Communications formulated and announced the Ubiquitous Network Society R&D Program II. Research on ultra-realistic communication was established as a major theme, with 3D imaging technology a special priority.

Ultra-realistic communication should further a creative, active lifestyle and universal participation in society. We believe it has the potential to help address social issues, environmental problems, and education. It could also be useful in developing new markets and contributing to expanding economies and industries. To this end, we intend to increase our efforts in research and development on ultra-realistic communication through collaboration between industry, academia, and government at the URCF and through public outreach. In particular, by 2011 NICT plans to develop an experimental electronic holography system that will take a 3D color moving image and display it in real time on a 2in-diagonal screen with a 15-degree viewing angle.