Protein-based optical filters for image processing
A receptive field (i.e., a limited field of view) is an important functional unit for processing visual information. If the shape of a receptive field is understood, it can be used to predict the time response of a cell to arbitrary spatiotemporal stimulation patterns. Although silicon-based artificial receptive field devices have previously been proposed, these have been only partially successful and they do not satisfy requirements for simplicity and resolution.1 New materials for such devices therefore need to be developed.
In humans, visual information is first processed within retinal ganglion cells (RGCs). The information is then transmitted to the lateral geniculate nucleus (LGN) of the brain, and finally to the primary visual cortex in the cerebrum. Indeed, certain sensory neurons have receptive fields. For instance, neurons in RGCs and the LGN have coaxial-shaped, center-surrounding receptive field structures that are composed of excitatory and inhibitory regions. An on-center receptive field, in which the center is sensitive to bright stimuli and the surrounding region is sensitive to dark stimuli, is shown in Figure 1(a). It is possible to model the shapes of such receptive fields with the Difference of Gaussians (DOG). Cells in the visual cortex—unlike those at earlier stages in the visual path—respond much better to a moving line than to a stationary line. These ‘simple cells’—see Figure 1(b)—have spatially oriented receptive fields with elongated alternating excitatory and inhibitory domains.2 In addition, they can be described accurately by a Gabor function,3 which is simply a Gaussian-modulated sinusoid. The DOG and Gabor filters are regarded as an important tool for a variety of image processing activities (e.g., edge detection, image enhancement, and pattern recognition).
In some previous experiments,4–7 it has been reported that protein-based artificial receptive fields can yield edge-detection capabilities. In our work, we have thus developed a new artificial receptive field with the use of a photoinduced protein that can simulate both excitatory and inhibitory responses.8 The unique protein that we use—bacteriorhodopsin (bR)—is found in the cell membrane of Halobacterium salinarum and is similar to the visual pigment rhodopsin, which is found within the rod cells of human eyes. When the bR is illuminated, it transports protons out of the cell membrane. The resulting pH change generates a transient photocurrent that lasts only for the instant of the light switching (i.e., on or off), and which exhibits a polarity reversal (i.e., positive or negative). In this work we propose the use of a ‘patterned’ 2D-Gabor filter (i.e., a description of a simple cell's receptive field) in addition to a 1D DOG filter. This builds on our earlier study in which we produced a simple motion sensor by patterning bR films on electrodes.9 In our work, we make use of mask patterns, which are easy to print—with the use of a graphics program and a laser printer—in desired patterns on transparency films. With these mask patterns we are able to dramatically improve the image processing capabilities of receptive fields.
The experimental configuration of our bR receptive field—see Figure 2(a)—is based on a model of a binarized 1D DOG function. Our receptive field consists of a bR dip-coated thin film, as well as an electrolyte that is sandwiched between glass plates with indium tin oxide (ITO) electrodes. We coat two rectangular patterns—with different size bR films—on the front and rear ITO plates, which correspond to the excitatory and inhibitory regions, respectively. In this setup, we also fill the electrolyte solution (potassium chloride, pH 8.3) with a spacer. In addition, we use a liquid-crystal display projector, with a metal-halide lamp, to display images onto the sensing area. Various input images are produced (and the scanning speed is controlled) using the presentation software Keynote. The responses of the edge and of the thin light bar (with a width of 1.3mm) when they are scanned across the sensing area of the 1D DOG filter are shown in Figure 2(b). For the edge case, we observe a zero-crossing profile when the position of the edge coincides with the center of the filter. Both of these results are in excellent agreement with the theoretically predicted responses of X-cells to various stimuli (i.e., a moving edge and a thin bar).10 These predicted responses are images convolved with a Laplacian filter, i.e., ∇2G (where ∇2 is a Laplacian operator and G is a 2D Gaussian distribution).
A 2D Gabor can be expressed mathematically in terms of the wavelength of the cosine function (λ), the variance of the Gaussian function (σ), and the spatial aspect ratio (γ). To make a binarized Gabor pattern, we chose values of 9.0mm, 4.5, and 1.0 for λ, σ, and γ, respectively. In addition, we set the intensity values to zero (i.e., less than a threshold value of 0.4). The structure of an on-center Gabor filter that mimics a simple cell receptive field is shown in Figure 3(a). In this structure, the surface of a bR 1D DOG filter is covered with a vertically directed Gabor pattern. In our experiments, we verified that our bR Gabor filter works well for line detection. We also scanned a thin light bar on the tilted filter to determine the selective orientation characteristics. We measured a maximum response at 0° and almost no response beyond 45°. These results therefore indicate that our bR Gabor filter only responds to spatial intensity changes along certain orientations.
Furthermore, we used a stripe with six bars to analyze the spatial frequency sensitivity for different bar widths (1–20mm). We are able to predict the preferred spatial frequency from the Fourier transform of a Gabor function. We measured peak responses at about 0.11mm−1, which corresponds to the reciprocal of λ equal to 9.0mm: see Figure 3(b). Our bR Gabor filter responds well to the predicted spatial frequency and therefore acts as a bandpass filter. If there is partial modulation in the spatial frequency of the input stripe, it leads to a significant decrease in the photocurrent intensity: see Figure 3(d). With this filter, we are therefore able to detect small differences of pitch (e.g., for a printed circuit) just by scanning projected images.
We have designed new receptive-field-type filters with the use of the photoreceptor protein bR, and we have experimentally shown that real-time image filtering is possible with the use of only one sensing element. We have also demonstrated that our DOG filter—which mimics on-center RGC receptive fields—has the function of a Laplacian filter and can act as an edge detector. In addition, our Gabor filter exhibits orientation and spatial-frequency selectivity, which is similar to that of simple cell receptive fields in the human primary visual cortex. Image processing with bR receptive fields is therefore similar to perception in the human visual system. The use of bR instead of silicon in image processing techniques can offer many advantages (e.g., visual functions at the material level, environmental friendliness, simple preparation processes, no need for additional external connections, and lower costs). We are now planning to explore the possibility of using our bR filters within digital image processing systems so that their operating times and power consumption can be reduced.
This work was supported by the Japan Society for the Promotion of Science Grants-in-Aid for Scientific Research (KAKENHI) program (grant 15K00226).
The University of Electro-Communications
Yoshiko Okada-Shudo is an associate professor. In her research she focuses on nanophotonics and biophotonics. In addition, her current interests include nonlinear optical imaging of membrane proteins.
Tokimasa Tanabe is a first-year MS student. He studies bioelectronics.
Takayuki Mukai is a second-year MS student. He studies bioelectronics.
Takuma Motoi graduated from the University of Electro-Communications in 2015. He is now working for Alps Electric Co., Ltd.
National Institute of Information and Communications Technology
Katsuyuki Kasai is a senior researcher. His research interests include bioelectronics and quantum optics.