Digital volumetric holograms for medical imaging

How did this technology, volumetric holograms, start for you folks? And how did it find its way into medical imaging?

Stephen Hart: I was at Imperial College, London, in the applied optics and astrophysics groups there. Our group had built an imaging, scanning Fabry-Perot interferometer for astronomical use, which could, in a very short period of time on a telescope, get, instead of a 2D image, a 3D volume of data. We had x and y on the sky and our z dimension was wavelength. We could map out the red and blue shifts of receding and approaching gases, respectively, in different astrophysical objects. I had all this 3D data on which I had done all the computer graphics things, but I still didn't understand this information.

Allan Wolfe: When I first met Stephen, they had the technology that enabled them to produce real volumes. We started thinking about the various potential applications and, for a variety of reasons, medical imaging (Figure 1) made sense to us. Over the last couple of decades, physicians realized that viewing and knowing everything that was in a volume was important, and the development of technologies like computer tomography (CT) and magnetic resonance (MR) made it possible to collect information about the volume. What remained was to look at the volume in some coherent way, because existing displays only showed the volume as a series of 2D pictures rather than a real 3D image.

Figure 1. In the process of operating on a large brain tumor in the region of the acoustic canal in a patient with neurofibromatosis, Dr. John Collins (Tripler Army Medical Center, Honolulu), inserted a cotton swab in the cavity where he was excavating the tumor. By directly comparing the volume of the cotton swab to the tumor in the lifesize hologram of the patient, he was able to determine that he had removed only 1/3 of the tumor.

Hart: In the astronomical objects I was looking at, we weren't quite sure what it was we should be seeing. Holography gave me a way to directly print those slices, one behind the other, in 3D, and actually see the structure of the objects. But, as an astronomer, we get a few data sets like that a year, whereas in radiology, something like 50 million times a year worldwide, a patient undergoes a CT or MR scan. As far as this technology is concerned, that sort of data is identical to astronomical data. It's parallel slices, all 2D, black and white images, and somebody, in this case, a radiologist or surgeon, is trying to understand what the structures are, how things interrelate, e.g., how blood vessels relate to tumor tissue.

When did you folks found this company?

Wolfe: It was 1988. Initially, we were trying to confirm if we had a viable product. In London, Stephen was using a lot of equipment in a kind of midnight engineering approach, making images on glass plates, which took a lot of time and care. In order for there to be a business, we needed an instrument that could do this in a reasonably short period of time and could be operated by someone who had the equivalent skills of a radiologic technologist. We needed to get the images onto film and, finally, reconstruct these images without having to use a laser. Stephen had worked that part out on a smaller scale while he was in London.

We set up a laboratory in a warehouse in Laguna Hills, California, and set out to confirm that these objectives were achievable. We got radiologists and physicians to look at these images and determine if they were of any value. Some thought that this warranted the Nobel Prize. Others thought it was totally useless because one could look at a series of 2D images and create, mentally, a volume of information. We didn't know which interpretation was correct. Initially, our images were somewhat crude and doctors were saying, "If this had more resolution or more gray scale, it might be useful." Once the images improved so that they had adequate resolution, the doctors started to focus on the content. Then it became abundantly clear to us that this was an important contribution and that we could build a business around it.

Hart: We were fortunate to have a good group of venture capital investors. We did a large number of clinical trials prior to trying to build a commercial version of the system. At the outset, we concentrated on placing only the white light display part of our hardware-which is low cost and doesn't contain a laser-into hospitals. We built all the expensive and complex parts here in the lab and proved that the technology was something radiologists and surgeons would benefit from. Only after that did we make the transition to produce a commercial version sized for hospitals and that had automatic features.

Figure 2. The camera system. (a) The schematic. Without an actual object to light, Voxel uses an LCD system to project the digital data of one slice from a patient's CT or MR scan onto a diffuse transmissive screen. The projected image of this single slice is the object, which the holographic film records. The screen is then moved farther from the film, and a second slice is projected and recorded. Because holography encodes distances as well as brightnesses, the individual slices replay at their different positions. Voxel's Voxcam repeats this process for every slice of the scan, typically superimposing 60 or more individual holograms on one holographic film.

Figure 2 (b). In this photo of a prototype, an Argon ion (1 W, 514.5 nm) laser beam is split into reference (left leg) and object (right leg) beams and recombined at the film, f (seen behind the collimating lens for the reference beam). The object in this system is an image of one slice of an MRI scan (brain) projected onto the diffuse screen, s, by the projector, p. Any one slice gives x, y, and brightness information. The screen is mounted on wheels to give z dimensional information correlated to the slice. The incoherence of the object beam from the slice and coherence of the reference beam together form the interference pattern at the film to give brightness and depth.

How does this work?

Hart: With a traditional hologram, you have the object, the object beam, and the reference beam, as your readers know. In our case, we don't have the object. What we do have is the digital data, one slice of patient data. So we feed that data into, in essence, a digital video projector and project that onto a diffusing screen so that it can be seen from different directions (Figure 2). We're actually making a full-aperture, off-axis, transmission hologram. We then move that screen and record again and again. If we have 60 slices, we move that screen to each of 60 discrete positions and at each position, record a hologram of the correct, appropriate slice.

With our hologram, there's no compromising the depth cues you get from our images. It's just like looking at a real object hanging there in space. That's the key. No distortions, no deletions.

Then, for the display, all we need is white light (Figure 3).

Figure 3. The Voxbox display uses controlled light and dispersion compensation to replay a transmission hologram so that the recorded data is replayed accurately. White light from a small tungsten-halogen lamp is collimated with a large acrylic Fresnel lens, diffracted at an angle to match the hologram's reference angle, and dispersed to cancel the inherent strong dispersion of the hologram. A proprietary louver material between the grating and the hologram blocks residual zero-order white light while letting the grating's dispersed spectrum pass through to the hologram.

Tell me about image fusion, such as fusing CT and MR data.

Wolfe: In such a case, we're actually looking at two multiple exposure holograms, one from CT data and one from MR data. One of the unique things about these multiple exposure holograms that's hard to appreciate until you see them in person is that while you can see all of their contents, they're also transparent. So you can see through them. If you've made a CT examination of a patient's brain, for example, and an MR of the patient's brain, so long as you scale them both the same way, when you lay one on top of the other, you see all of the information that is in both of them. That enables you to take advantage of the fact that certain scanning techniques "see" certain kinds of tissue better. For example, the physician who wants to see a tumor that is easier to discern in MR in relationship to a bone in the skull, which is easier to see in CT, can accomplish this simply by stacking a couple of holograms.

So, you can even use images from an ultrasound then?

Wolfe: Exactly, we have done ultrasound. We've done nuclear scanners. So long as the information comes along as a series of slices, we can produce a hologram.

Hart: Each hologram is made from digital data, typically from a medical scanner. But if we have digital data from an astronomical instrument, a confocal scan microscope, or anything that gives us 2D digital data-a stack of them at different depths through whatever the target is-we can make a hologram of it.

So far, we've made about 6,500 or so holograms of human patients, several holograms of astrophysical nebulae, and several holograms of optical microscopy data. We've straddled the universe from the smallest to the largest.

You use a digital projector?

Hart: We're using a poly-silicon LCD panel as you would use in a regular video projector for looking at your business graphics or video. The output from a modern CT or MR scanner is digital images flowing across an Ethernet connection. Each image is typically a 512 square matrix and each pixel has a gray scale value. That image data has been acquired and formed by the scanner. We're not involved in acquiring or forming the data. The image data we're given we can look at on the CRT, just as if we were looking at it on a desktop computer. Or we can project it. In our case, we project it onto a diffuse surface, which provides the object for the hologram.

Are your films different from regular holographic film?

Wolfe: Yes. That's one reason why this business is so exciting to us. Let's take the last year for which we have data, 1995. There were 26 million CT and MR examinations performed in the United States then. They were performed at approximately 5,000 different locations that have CT, MR scanners, or both. While selling the hardware to produce these holograms would be nice, eventually you would saturate the market . . . and it's over. One of the things that is proprietary to the company is the film itself. Each time people make a Voxgram, the company sold them the film on which they made the hologram. They cannot use standard radiology film or film that they can buy from any other supplier.

Hart: We made the Voxgram film the standard size that radiologists are used to-14 X 17 inches, like a chest x-ray. There were a lot of issues in commercializing this. We needed to be able to transport the film automatically so that the technologist running the printer doesn't have to handle film. It's automatically dispensed from a film dispenser and automatically processed. I'm working to have the best combination of film, chemistry, and all the rest of the technology so this can be done rapidly. The film is machine processed in a processor docked to the print engine. The film handling is completely automatic; every part of the system is picked and matched together so that the process is routine.

Wolfe: As far as the user is concerned, they take raw film and load it in one end of the camera and finished holograms come out the other end.

What are the advantages of the Voxbox, your 3D holographic system?

Wolfe: These holograms differ from the kinds of medical images people are used to seeing. Voxgrams have seven special characteristics: transparency, perspective, context, fusion, interactivity, faithfulness, and life size.

Transparency produces another effect called image fusion, but transparency by itself is important. If we examine the chest cavity, we want to see the ribs, the heart, and the spine. Because these images are transparent, you can look at any object and yet look through any object, simply by refocusing your eye. For example, you can see your own reflection in the glass pane of a shop window and then refocus to see what's inside.

Our technology works exactly the same way except that we're looking at or through 60, 80, or 100 slices. It is the only technology that enables a physician to look at the whole volume at once. It's not simulated 3D, as you sometimes find on a computer screen.

Perspective is the ability to understand that when you look at a volume, what you see varies depending on where you're standing relative to the volume. If I'm at the left side of the chest looking in, the shape of the heart is different than when I'm at the right side of the chest looking in. I can see more, simply by changing the relationship between myself and this image.

Context is the ability to appreciate that the objects inside the image have a relationship to each other, which one couldn't appreciate if one were not looking at the whole volume. For example, imagine the chest cavity with a sphere close to the back and one close to the front. By looking at the hologram, you not only know you have two spheres, but you know where they are relative to the chest and relative to each other.

We've talked about image fusion.

Interactivity is a very critical and important attribute of these holograms for surgeons, particularly, neurosurgeons. Because these images are real volumes, you can put real objects into them (Figures 1,4,5,6). If you want to determine what instrument to use to clip a blood vessel that is in danger of exploding, or if you want to determine what size the prosthetic hip ought to be that you're going to put into a patient, you can take the real object and put it into the image and check the fit.

Next is faithfulness. We do not have to alter these data sets in order to produce these volumes. Other approaches at depicting volume involve computer interaction with the data to create the illusion that one is looking at the volume, either by spinning it or shading it, relighting it, etc. With ours, the doctor can have absolute confidence that what he's looking at is what the scanner scanned, without us performing any operation on it.

The last trait is that our images are life size. Normally, these images are small when they're reprinted as 2D; they're too unwieldy otherwise.

All of these characteristics, with the exception of life size, are unique to these multiple exposure holograms and do not exist in any other medical display system.

Figure 4. A bone screw inserted into a hologram of the cervical spine of a patient demonstrates the unique interactive characteristic of multiple exposure holograms-Voxgrams. In one paper, Dr. Edward Benzel (University of New Mexico), demonstrated that bone screw placement with 3D holography was as accurate as fluoroscopy and twice as rapid.

Figure 5. Volumetric hologram of a knee from CT data. Notice the pointer under the patella, or kneecap, showing once again the interactivity of the physician to the image- i.e., the pointer is within the volume of the image.

Figure 6. This shows the fusing of two skull holograms from the same data. (a) The underlying film shows the top half of the skull, which is projected into the holographic lightbox. The film on top is the bottom half of the skull and is projected out of the lightbox. If both films are flipped about the horizontal axis, the top half of the skull will now be projected out of the voxbox, the bottom half projected into the voxbox, and front and back of the skull will be reversed top to bottom. (b) This fused image, with a pointer going into the a child's skull from the bottom, shows the multiple plates and screws used to stabilize the cranium of a patient with craniosystosis, a skull deforming condition (image provided by Dr. Jill Hunter, Children's Hospital of Philadelphia).

Can you vary the size of these Voxgrams?

Wolfe: It's possible now. We use life-size because that's what the doctors prefer today. We can scale these images down or enlarge them. We've, in fact, made some holograms of microscopic objects, including the bone, which are quite dramatic. I feel people will use those applications as time goes by.

Hart: The smallest object we've recorded a Voxgram of was, I think, about 20 µm across. It was a spirogyra, an algae.

Wolfe: We don't have any limits. As long as the data are there, we can make the hologram.

Of course, resolution is everything.

Hart: It is the full resolution from the scanner data.

Wolfe: 512 X 512 pixels in a CT and 256 X 256 in an MR, per slice.

Hart: If you try to get much higher resolution than that, you're on the scale where the patient is actually moving. So, you'd have to go very fast as well. And high speed, high resolution scanners are extremely exotic.

Wolfe: Some physicians confused resolution with the process. When you're looking at the whole volume, you're actually seeing what it really is. When you slice it up, by virtue of slicing it up, you're creating an artifact. Think of an onion. If you look at an onion, it has a shape, it has a consistency, etc. When you slice an onion, it looks like it has a lot of rings. It really doesn't have rings; it has spheres. By creating the slice, you've made the spheres into rings.

Sometimes, when a doctor looks at the whole volume, he says, "God, I don't see this line." That line didn't exist except as an artifact of the slice. In order to convince people some years ago, what we did was produce single slice holograms that were equivalent to the information in the single slices they were used to looking at and they readily admitted we had all the resolution there was.

Where is the technology now? When will it go to market?

Wolfe: We were hoping to have it on the market by now. We had made an arrangement with a company in Massachusetts to design and manufacture the camera part of this system. The film is made for us by Konica. And a local vendor produces the viewer. The camera that has been produced hasn't met our standards. Frankly, we've been having a little contretemps with the vendor because the instrument doesn't consistently produce holograms as good as the thousands we've produced here. We expect to resolve those differences over the next several weeks or next couple of months, and we expect to have systems in the field early this year. But we're not exactly sure when.

There is a system in New Mexico and that is where we confirmed that the performance of the camera did not meet our standards. Some people have urged us to go ahead and deliver what we have, but we think it would be a mistake to compromise the quality of these holograms in the initial systems. So, we're not going to do that.

How much will these cost in relation to the CT and MR scanners?

Wolfe: One way to look at it is this way. The typical cost of a CT examination ranges from $350-$1000. The typical cost of an MR examination ranges from $500-$1800. Adding our display to the examination, even accounting for the amortization of the equipment, adds $60-$70 to that cost.

Hart: That was a key point. This is intended to be a routinely used technology for all manner of scanners, all manner of scans, all manner of patients, all manner of pathologies. We composed the technology from parts in such a fashion that it is not a costly, exotic, special-purpose, special-application tool that you would only use in certain cases. It's intended to be something that you can use routinely. And, we've demonstrated that it is routinely useful.

What do you see for the future?

Wolfe: Well, I think that eventually a substantial proportion of all the examinations that are performed will be displayed as holograms. So, I think our business will grow very dramatically. At some point in time, being able to put these holograms in motion would be a useful feature if the scanners get fast enough to collect the information while it is in motion.

We're going to make some kinds of examinations safer and easier. For example, if scanners were fast enough, you could inject dye in a peripheral blood vessel and produce holographic angiograms of the heart.

How about airport baggage inspection?

Hart: We're not quite fast enough as of today. From the time the data is available to the time the operator has a film at hand, it's about half an hour. That's fine in a hospital, but for baggage inspection, you want a few seconds, perhaps half a minute for each piece. It may be a future application.

Wolfe: One application we're looking at is the data that are collected to estimate oil and gas reserves. It turns out to be a series of slices and that looking at the volume may be more edifying than the way they are looking at the data today. So, we've begun a slight effort. But, being a little company, we need to establish ourselves medically before we pursue these other applications. But we have high hopes and aspirations.

Allan Wolfe, M.D., has two decades of medical industry and venture capital experience. He has been the President and CEO of VOXEL since its inception while maintaining an active, investment-oriented interest in emerging medical companies.

Stephen Hart helped found Voxel to commercialize multiple-exposure display holography. Prior to this he founded a holographic imaging company in London, England, and performed research in astrophysics and holography at Imperial College of Science, Technology, and Medicine.