Mosaicing of microscope images in the presence of large areas with insufficient information content

Transcription

1 Mosaicing of microscope images in the presence of large areas with insufficient information content Yulia Arzhaeva and Changming Sun CSIRO Mathematics,Informatics and Statistics, Locked Bag 17, North Ryde, NSW 1670, Australia ABSTRACT In virtual microscopy, multiple overlapping fields of view are acquired from a large slide using a motorized microscope stage that moves and focuses the slide automatically. A virtual slide is reconstructed by combining digitally saved fields of view into an image mosaic. A seamless reconstruction requires the correction of unknown positioning errors of the stage. This is usually done by automatically estimating alignment parameters of the tiles in the image mosaic. But finding accurate alignment parameters can be inhibited by the presence of tiles that lack information content in the areas of overlap. In this work we propose a new mosaicing method that accesses information content of each overlap and performs pairwise registrations of adjacent tiles only if the content of their overlap is deemed sufficient for successful registration. For global positioning of tiles an optimal stitching path is found by tracing such content-rich overlaps. We tested the proposed algorithm on bright field and fluorescence microscope images and compared the results with those of an existing algorithm based on simultaneous estimation of global alignment parameters. It is shown that the new algorithm improves perceived image quality at boundaries between tiles. Our method is also computationally efficient since it performs no more than one pairwise registration per tile on average. 1. INTRODUCTION Virtual microscopy (VM) is enabled by computer controlled microscopes. In VM, microscope slides are scanned and stored as digital images. Following this, slides can be browsed on a personal computer and shared over a computer network. Furthermore, various automated image analysis techniques can be applied to large digital databases of slides thereby propelling discoveries in biomedical research and drug development. When microscope slides are too large to be acquired as single images, multiple overlapping fields of view (FOVs) are acquired and composed into a large single image. A sequential scanning process is assisted by a fully automated motorized microscope stage that moves and focuses the slide. The reconstruction of the original slide is performed by a software module that aligns acquired FOVs with their adjacent FOVs. This operation is called image mosaicing or stitching, and images representing individual FOVs are called tiles. Mosaicing is an essential preprocessing step for large microscope slides before any image analysis could be applied to them. An image stitching algorithm should be able to align tiles so as to minimize visual artifacts at their borders. There are two sources of artifacts - unavoidable positioning errors of the motorized stage, and uneven lighting within and across images. These artifacts are evident in Figure 1 which demonstrates a mosaiced histology section, stitched using uncorrected tiles positions obtained from an acquisition protocol. Therefore, a stitching algorithm should include geometric correction and intensity correction procedures. Stitching algorithms are widely used in computer vision, for example, in creation panoramic photo-mosaics. Szeliski 1 presents an extensive survey of geometric alignment and blending techniques. In this paper we focus on optimal positioning of tiles in a virtual microscope slide. A general approach to geometric alignment consists of two steps. In the first step, adjacent tiles are locally aligned, i.e. matching points are found, or image registration parameters are computed between two tiles. Based on pairwise registrations, the absolute positions of tiles in the mosaic are estimated in the second step. Such an estimation is carried out sequentially or simultaneously, by solving a global optimization problem. In contrast to stitching panoramas, algorithms for stitching microscope slides can benefit from preliminary knowledge about an acquisition setup. It includes positioning of the motorized stage for scanning each FOV. Based on the position information, adjacent tiles are easily identified and approximate overlap regions between neighboring tiles can be estimated. The

2 Figure 1: Image mosaic of the 5 5 histology section without geometric and intensity corrections. transformation between a pair of adjacent tiles is constrained to translations in X and Y directions, due to constrained movements of the motorized stage. Publications on image stitching for 2D and 3D virtual microscopy and available software solutions have been extensively surveyed in Emmenlauer et al. 3 Most of the approaches rely on knowing scanning positions of the stage or manual pre-alignment of tiles in order to initialize the search of local correspondences between tiles. The work of Emmenlauer et al 3 is an exception where fast registration of tiles is performed as a pre-processing step, in order to identify adjacent tiles and find initial transformations between tiles. The absolute positions of tiles are usually found either by simultaneous optimization of the global transformation parameters of all tiles, 2, 3 or by graph-based methods, 4, 5 or using dynamic programming. 6 Simultaneous optimization of global transformation parameters has the advantage of an even distribution of alignment errors over all transformations. However, if an overlap between two tiles contains no or little information, a local transformation between them cannot be reliably found, which is likely to lead to a suboptimal solution of the global optimization problem. Microscopy images, such as images of neurites, histology sections, images of bacterial colonies etc, often contain relatively large background areas. Therefore, there is always a likelihood that a pair of tiles overlaps along such an area, or a whole tile contains nothing but noise. In Emmenlauer et al, 3 the problem of large areas with little or no information content is tackled by dropping tiles with unreliable local transformations out of the set of optimization equations and asking for a manual placement of such tiles in the end of the processing. An automated solution for this problem has been proposed by Steckhan et al. 4 Their method is inspired by a general graph-based approach to image stitching. 7 In their paper, a graph is constructed based on the results of pairwise alignment of tiles, and global alignment is obtained through a search for an optimal, minimum cost path in the constructed graph. In this way, unreliable local transformations resulting from empty areas are not taken into account. In our paper we present a similar automated approach. Our work differs from Steckhan et al 4 in that for Some authors also consider rotation, possibly caused by misalignment of the microscope camera sensor and stage axis, as well as mechanical errors 2

3 pairwise image registration we use a metric based on mutual information which is better suited for registering images with unknown intensity variations due to lightning artifacts than phase correlation used by Steckhan et al. Another advantage of our method is a selective computation of local registration parameters, because image registration is a computationally costly procedure. After the assessment of the information content of each overlap, an optimal path for stitching tiles is found based on the results of this assessment. Local registration is performed only between adjacent tiles that lie on the stitching path. We have devised a custom algorithm to search for an optimal stitching path, which is an alternative to graph-based search algorithms. 2. MATERIALS Two sample images used in this paper were acquired with an Olympus BX61 microscope (Olympus, Tokyo, Japan) equipped with a motorized precision stage and 20 Olympus UPlanFL objective lens. The stage positioning accuracy was 3 µm. The images were acquired with two different cameras. A Media Cybernetics Evolution QEi camera (Media Cybernetics, Silver Spring, MD) was used to capture an images of fluorescently labelled bacteria, and a QImaging MicroPublisher 3.3 RTV colour camera (Burnaby, BC, Canada) was used to capture a hystology section. A histology section contained 5 5 FOVs, with a tile size of and only 5% overlap between tiles. The fluorescent microscopy image also contained 5 5 FOVs, with a tile size of and 20% overlap. Both images were scanned by rows from top to bottom, with individual rows scanned from left to right. The same images were used for evaluating the stitching algorithm in Sun et al Estimation of information content 3. METHODS At first, the information contents of all overlaps are accessed. An overlap is extracted from either of two adjacent images based on approximate translation parameters obtained from the motorized stage. As a measure of information content, a local coherence is derived from the structure tensor 8 at each point: c = { ( λ1 λ 2 ) 2, λ 1+λ 2 if (λ1 + λ 2 ) > 0 0, otherwise, where λ 1 and λ 2 are eigenvalues of the structure tensor. (1) The coherence ranges from zero to one. It provides a measure of the coherence of the oriented structure in the vicinity of the point. Isotropic areas are characterized by zero coherence, while in the vicinity of edges the coherence is close to one. For each overlap, the fraction of points with large coherence values is computed. If this fraction is lower than a user-defined threshold, an overlap is considered non-informative. Otherwise, an overlap is considered informative. 3.2 Optimal stitching path We assume that tiles that have an informative overlap can be registered reliably and, therefore, aligned to each other accurately. Our goal is to connect all the tiles in the mosaic using only reliable local transformations. Therefore, if a tile has at least one informative overlap with its neighbors, it is stitched to the mosaic through one of them. Here, an algorithm is presented that finds a stitching path through informative overlaps and computes the tiles absolute positions in the mosaic. Pseudocode of the algorithm is given in procedure ComputeAbsPositions in Figure 2. The absolute position of each tile in the mosaic is derived from the absolute position of an adjacent tile and a transformation between them. Since the movement of the motorized stage is translational, a possible transformation between two adjacent tiles is limited to a translation. The algorithm is run iteratively until an absolute position for each tile is found. In each iterations, all unconnected tile are visited. If a tile has a neighbor which absolute position is already known and there is an informative overlap between them, the absolute position of this tile is computed using an exact translation between the two tiles. Exact translation parameters are found via image registration described in Section 3.3. To initialize the algorithm, the absolute position of one of the

4 two tiles with the most informative overlap is set to (0, 0). This is an overlap with the highest fractions of points with large coherence values. In the beginning of the algorithm, pairwise translations Tinit x and T y init are computed for the two pairs of tiles with the most informative horizontal and vertical overlaps. Tinit x and T y init are computed for two purposes: in order to initialize image registration for other pairs of tiles, and to be used for stitching those tiles that have no informative overlaps with any of their neighbors. Appropriate initial transformation parameters ensure that registration almost always converts to an accurate solution. To find Tinit x and T y init, image registration is initialized with approximate translation parameters obtained from an acquisition setup. Sometimes, a group of informative tiles is connected to the rest of the mosaic only through non-informative overlaps or tiles. Such an isolated island can be stitched to the mosaic only if the rule of not connecting a tile via a non-informative overlap if there is at least one informative one is broken. It is enough to break this rule once in order to stich the whole island. Procedure ConnectIsolatedTiles in Figure 2 describes connecting an island. When all the tiles absolute positions are computed, they are shifted to the non-negative range, if necessary. procedure ComputeAbsPositions(S) compute T x init, T y init ; set a (0, 0) tile; while S is not empty do /* S = unplaced tiles */ D = {} ; /* S i,inf = neighbors of tile i with informative overlap */ n = size(s); /* S i,inf = neighbors of tile i with non-informative overlap */ foreach i S do if S i,inf is empty then if j S i,inf so that AbsP os(j) is known then AbsP os(i) = AbsP os(j) ± T init ; /* T init is either T x init or T y init */ delete i from S; else if j S i,inf so that AbsP os(j) is known then compute T exact between i and j via registration; AbsP os(i) = AbsP os(j) ± T exact; delete i from S; else add i to D; if size(s) == n then /* if no tiles were connected in for-loop */ ConnectIsolatedTiles(D); shift all AbsP os(i) to the positive range; end procedure ConnectIsolatedTiles(D) Resolved = false; k = 0; while!resolved do i = D[k]; if j S i,inf so that AbsP os(j) is known then AbsP os(i) = AbsP os(j) ± T init; Resolved = true; delete i from S; else k = k + 1; end Figure 2: The algorithm for computing the absolute positions of tiles. 3.3 Pairwise registration A pair of overlapping tiles that lie on a stitching path and have an informative overlap is aligned using a translation. Translation parameters are found by maximizing similarity between two tiles. For this purpose,

5 mutual information was used as the cost function. 9 Mutual information measures how much information image intensity in one image tells about image intensity in the other image. The actual form of the dependency does not have to be specified. It makes the cost function stable to intensity variations between tiles caused by lighting artifacts. To optimize the cost function, a regular step gradient descent algorithm was applied. 4.1 Implementation details 4. RESULTS Pairwise registration was implemented using a registration framework from an open-source library of software tools for image analysis, ITK. 10 User-defined thresholds for estimating the information content of an overlap were set as follows: an overlap between adjacent tiles was considered informative if not less than 1% of pixels in the overlap had a coherence value larger or equal to In addition to geometric alignment, an intensity correction procedure was performed on the test images. To this end, optimization of intensity transformation parameters was implemented as described in Sun et al. 2 Times required for mosaicing of the histology section and the fluorescent bacteria image were 43 and 21 sec, respectively, including the intensity correction time. Computations were performed on a PC with a 1.83 GHz Intel CPU and 3 GB RAM. 4.2 Visual evaluation For both test images visually consistent mosaics were obtained. In Figure 4 the resulting mosaic of histology images is shown. The stitching path found by the proposed algorithm is marked by fine white lines connecting the approximate centers of tiles. Tiles connected by dashed lines do not have neighbors with informative overlaps, therefore they were stitched using the initial translation parameters, Tinit x or T y init. Figure 5 shows two close-up views of the histology mosaic alongside the same views from a mosaic obtained by applying the global optimization method by Sun et al. 2 In these examples, our method produced a better alignment. Similar improvements were found in the mosaic with fluorescently labeled bacteria. This mosaic is shown in Figure??. 4.3 Quantitative evaluation Usually, there is no ground truth available for quantitative evaluation of stitching algorithms. In order to evaluate the accuracy of the proposed algorithm, we simulated the scanning procedure of the motorized stage. 14 tiles from the histology section, where content occupied at least half of a tile area, were cut into 5 5 subimages with 20% overlap. The cutting positions were randomly varied around the projected cuts to imitate the positioning accuracy of the stage. For each subimage, its positioning error was computed as the average of the Euclidean distances between the reference translation vector to every adjacent subimages and the translation vector obtained by the stitching algorithm. Translation vectors to adjacent subimages with non-informative overlaps were excluded from the computation because their accuracy are not vital. We obtained a mean error of 0.13 pixels and a standard deviation of 0.23 for 333 subimages that had at least one informative overlap with adjacent tiles. 5. CONCLUSIONS In this work, a new algorithm for mosaicing of microscope images is presented. It takes into account a possible presence of non-informative areas in a virtual slide and avoids estimating precise alignment parameters in such areas. The performance of the algorithm was evaluated visually on two real slides, with visually consistent results. Quantitative evaluation on an artificial example demonstrated the accurate alignment of tiles in the informative areas of a slide. 6. FUTURE WORK This work could be improved by using a more consistent measure of information content. Currently, the local coherence that we employ has no direct relationship to any local registration method. We would like to employ a measure with a strong ability to predict the success of local alignment. Besides, in the absence of simultaneous optimization of global alignment parameters we risk to accumulate a displacement error in case one of local translations is wrong. Therefore we are developing a modification of the proposed method. It uses feature

6 Figure 3: Image mosaic of the 5 5 histology section with corrected alignment and intensities. The stitching path, connecting the centers of tiles, is marked by white lines. Dashed lines denote stitching through non-informative overlaps. Areas marked by A and B are shown close-up in Figure 5. Figure 4: Image mosaic of the 5 5 fluorescently labeled bacteria with corrected alignment and intensities. A stitching path is not shown to avoid obscuring fine image details.

7 (a) Horizontal stitching (b) Vertical stitching Figure 5: Close-up views of stitched tiles from the histology section. Images in the left column were aligned using the global optimization method. 2 Images in the right column were aligned using the proposed method. The white arrows point out to discrepancies along the seam line between two tiles. points detection and matching between adjacent tiles. Based on the results of matching, a conclusion is drawn whether a pair of tiles possesses an informative overlap. Also, the results of matching are used in the stitching algorithm described in this paper to find the absolute positions of tiles. The tiles positions are further refined by solving a global optimization problem. Pilot experiments with the modified algorithm showed very good results in stitching images of neurons. REFERENCES [1] Szeliski, R., [Image Alignment and Stitching: A Tutorial], Now Publishers, Inc. (2006). [2] Sun, C., Beare, R., Hilsenstein, V., and Jackway, P., Mosaicing of microscope images with global geometric and radiometric corrections, Journal of Microscopy 224, (2006). [3] Emmenlauer, M., Ronneberger, O., Ponti, A., Schwarb, P., Griffa, A., Filippi, A., Nitschke, R., Driever, W., and Burkhardt, H., XuvTools: free, fast and reliable stitching of large 3D datasets, Journal of Microscopy 233, (2009). [4] Steckhan, D. and Wittenberg, T., Optimized graph-based mosaicking for virtual microscopy, in [SPIE Medical Imaging: Image Processing], Pluim, J. and Dawant, B., eds., 7259 (2009). [5] Chow, S., Hakozaki, H., Price, D., Maclean, N., Deerinck, T., Bouwer, J., Martone, M., Peltier, S., and Ellisman, M., Automated microscopy system for mosaic acquisition and processing, Journal of Microscopy 222, (2006).

8 [6] Appleton, B., Bradley, A., and Wildermoth, M., Towards optimal image stitching for virtual microscopy, in [Digital Image Computing: Techniques and Applications], (2005). [7] Kang, E.-Y., Cohen, I., and Medioni, G., A graph-based global registration for 2D mosaics, in [International Conference on Pattern Recognition], (2000). [8] Jähne, B., [Spatio-Temporal Image Processing: Theory and Scientific Applications], Springer (1993). [9] Thévenaz, P., Blu, T., and Unser, M., Image interpolation and resampling, in [Handbook of Medical Imaging, Processing and Analysis], , Academic Press (2000). [10] Ibáñez, L., Schroeder, W., Ng, L., and Cates, J., [The ITK Software Guide], Kitware, Inc. (2005).