A power stitching tool
For time-based studies, researchers need to repeatedly image plates, for example, of cell cultures. However, an optical microscope has a field of view smaller than a typical plate. It therefore generates a grid of overlapping partial images, and a stitching tool assembles a composite image, which may have tens of thousands of pixels per side. Stitching one such large image mosaic taxes the computing capacity of a high-end workstation. This becomes overwhelming when stitching repeat images taken over time.
Most high-end microscopes come with their own image-stitching tool. However, these tools are often manufacturer-specific and require fully calibrated equipment, a condition that is not always realized. An alternative is to use a manufacturer-neutral computational tool for image stitching. There are two broad sets of approaches for image stitching, known as feature based1 and Fourier-transform based.2
Feature-based approaches identify matching features in adjacent images and use these features to determine the extent of the overlap within the image pair. They will then use the overlap to transform the images and compose them into a single one. These approaches have gained widespread usage in consumer-oriented digital photography under the label of ‘panoramic photography’ because they are fast and because consumers almost always take feature-rich pictures. The main drawback of these techniques is that they must use a ‘segmentation method’ to detect features of interest in images.
By contrast, approaches based on Fourier transforms are more compute-intensive, but can reliably handle feature-poor images. Furthermore, the computational workload is more predictable as it does not depend on the presence and detection of features in images.
Optical microscopy often has to process feature-poor images (e.g., sparsely populated cell cultures) and may be used to derive measurements (quantitative information such as counts, densities, intensities, and so forth) from such images. We have developed a new stitching technique that minimizes the uncertainty between overlapping areas caused by a lack of features, while being nearly 500× faster than the leading alternative tool in the field.
The first step in image stitching computes translations between adjacent images using the Fourier-based phase correlation method. This produces a relative displacement and a cross-correlation value for each of the tile's neighbors. We use cross-correlation as a measure of the translation computation accuracy. Two adjacent images with enough texture in their overlapping area will have a cross-correlation value close to 1. In contrast, the cross-correlation value will be close to 0 when the overlapping area mostly consists of background or noise.
The second step assembles the tiles into a mosaic based on the computed translations. However, these translations cannot be used directly as they form an overconstrained system of equations. Our approach resolves this overconstraint by considering the system as a graph and progressively coalescing strongly connected components. We use the cross-correlation values as a measure of connection strength. This optimization minimizes the uncertainty and increases stitching accuracy. A blending algorithm is applied on the overlap area to create a smooth mosaic image (see Figure 1).
Our multithreaded implementation takes advantage of all the cores in a central processing unit and carefully manages memory to increase speed by two orders of magnitude over the leading alternative tool. Our implementation, devised for a graphics processing unit, accelerates the computation by another 5× for a total of ∼500×. Both implementations aggressively minimize the memory footprint of the computation so it can mosaic a 50×50grid in 8GB of RAM. They also generate image pyramids on the fly and allow a user to visualize their mosaics almost instantaneously (see Figure 2). These tools will be released as open-source plug-ins to ImageJ.3
The new image-stitching tool handles terabytes of optical microscopy data and minimizes uncertainty in image-stitching computations. The tool mosaics a plate's images and visualizes it at near interactive rates, thereby allowing scientists ready access to their results and giving them ample time to intervene in long-running experiments. This promises to start transforming experiments into computationally steerable ones. We are currently deploying our stitching tool internally to process large data sets and are readying it for release as a plug-in to the image analysis tool ImageJ/Fiji.4 We plan to explore integrating our tool with the open-source microscopy controller Micro-Manager. We will also investigate additional rotational corrections for image stitching in 2D and extending the approach to handle 3D images.
This work has been supported by the National Institute of Standards and Technology (NIST). We would like to acknowledge the team members of the computational science in biological metrology project at NIST for providing invaluable input to our work.
Joe Chalfoun is a research scientist. He received his doctoral degree in engineering from the University of Versailles, France, in 2005. He joined NIST in 2007. His interests are in medical robotics and cell biology.
