Transcription

1 1 Three-Dimensional Microscopy Data Exploration by Interactive Volume Visualization Shiaofen Fang Yi Dai Frederick Myers Mihran Tuceryan Department of Computer and Information Science Indiana University Purdue University Indianapolis Kenneth Dunn Department of Medicine, Division of Nephrology School of Medicine, Indiana University Address correspondence to: Shiaofen Fang Department of Computer and Information Science Indiana University Purdue University Indianapolis 723 West Michigan Street, SL 280 Indianapolis, IN phone: è317è fax: è317è Keywords: 3D microscopy, visualization, volume rendering, texture mapping, image processing.

2 2 SUMMARY This paper presents a new volume visualization approach for 3D interactive microscopy data exploration. Due to its unique image characteristics, 3D microscopy data is often not able to be eæectively visualized by conventional volume visualization techniques. In our approach, microscopy visualization is carried out in an interactive data exploration environment, based on a combination of interactive volume rendering techniques and imagebased transfer function design methods. Interactive volume rendering is achieved by using 2D texture mapping in a Shear-Warp volume rendering algorithm. Image processing techniques are employed and integrated into the rendering pipeline for the deænition and searching of appropriate transfer functions that best reæect the user's visualization intentions. These techniques have been successfully implemented in a prototype visualization system on lowend and middle-range SGI desktop workstations. Since only 2D texture mapping is required, the system can also be easily ported to PC platforms.

3 3 1 Introduction In the past twenty years microscopy has come to play anincreasingly important role in the study of cell biology. The advances in biochemistry and molecular biology have generated an increased appreciation of the cellular organization of the biochemical components of a cell. Although advances in confocal microscopy and image deconvolution have made it feasible to collect high resolution ëdunn et al. 1994, Shaw 1995ë, 3D image volumes of thick samples such as epithelial cells, application of this technology to 3D imaging is still in its infancy. Indeed, the proliferation of 3D microscopy in cell biology has generated vast amounts of image data that have not been suæciently explored and analyzed. The rapid advances in computer graphics and visualization, in particular volume visualization, have provided a great potential for the visualization of 3D microscopy images. Volume visualization is a new 3D computer graphics technique that is concerned with the abstraction, interpretation, rendering and manipulation of large volume datasets. Volume rendering algorithms, for instance, can directly display the entire volume dataset through semi-transparent images, and allow the viewer to peer inside the internal structures of the image volume for truly 3D data viewing and analysis. Although volume visualization methods and tools have been used in many scientiæc and medical applications, such as visual simulations and CTèMRI imaging, its applications in 3D microscopy are limited and largely ineæective. This is mainly because current volume visualization techniques are mostly designed for CTèMRI types of images, and are poorly suited for 3D microscopy applications. Several unique characteristics of microscopy data pose serious challenges to conventional visualization techniques. First, æuorescently-labeled samples characteristically have low signal levels, sometimes consisting of a single photon, so that microscopy images are typically much noisier than CT or MRI images. Furthermore, since excitation of æuorescence also destroys æuorophores through photobleaching, signal-to-noise ratio decreases with the collection of each focal plane of an image volume. Consequently, microscopy image volumes are usually very sensitive to

4 4 small changes in rendering parameters, such as the rendering transfer functions which map image intensity values to colors, opacities or shading parameters. Thus, ordinary volume visualization algorithms frequently fail to capture the delicate structures present in many cellular objects. Secondly, structures in the microscopic scale typically show higher complexity than those of the anatomic organs in CT or MRI images. This is particularly true in multi-parameter images, in which several diæerent proteins will be imaged simultaneously, each in a speciæc color of æuorescence. A third problem is that the structures of the objects to be examined are often partially or entirely unknown. This leads to the strong need for interactive navigation and searching capabilities in both the spatial dimensions and the transfer function space. Due to these special characteristics, 3D microscopy visualization is best performed in a data exploration environment in which users can interactively manipulate, search and render 3D microscopy images for their individual visualization goals. Two key technical requirements for such a visualization environment are interactivity and transfer function design. Data exploration is intrinsically a continuous and interactive process. Although many surface rendering and volume rendering algorithms have been developed ëlorensen and Cline 1987, Levoy 1988, Levoy 1990, Upson and Keeler 1988, Westover 1990, Lacroute and Levoy 1994ë, they have not been able to provide interactive rendering speed to support interactive data exploration. In fact, interactive volume visualization can currently only be obtained using either supercomputers ëparker et al. 1998ë or special hardware systems, which are not available on common desktop computers. The most popular hardware solution is using 3D texture mapping hardware which is only available on high-end graphics workstations èsuch as SGI's Onyx systemsè ëcabral et al. 1994, SGI 1998ë. A recently released hardware chip, VolumePro, by Mitsubishi also provides real-time volume rendering, but with considerable additional hardware cost. Furthermore, VolumePro does not support perspective projection, which is essential in data exploration applications. These limitations severely restrict the usability ofvolume visualization in 3D microscopy. Thus, our ærst goal is to develop a low-

5 5 cost technique for interactive volume rendering that uses only existing hardware features on common desktop computers. Another important component of data exploration is the searching for the right transfer functions that best reæect users' visualization intentions. The transfer function design problem is particularly important and diæcult with noisy and unfamiliar data sets, but has not received suæcient research attention. Most current visualization systems employ a trial-anderror approach, which is extremely diæcult and time-consuming for microscopy data. More importantly, visualization results obtained this way depend largely on the user's experience and ëluck", and can lead to confusing, misleading and dubious data interpretations. Previous eæort in improving the transfer function searching is rather limited. One approach is the evolution-based inverse design approach ëhe et al. 1996, Marks et al. 1997ë, which uses a stochastic search technique to generate many image samples based on an initial population of pre-deæned transfer functions, and then improves the samples based on the user's selections of the sample images at each evolution step. Although this approach provides some level of heuristics for transfer function searching, it is still a very time-consuming process, and does not support complicated or procedural transfer functions that cannot be represented by the pre-deæned function combinations. Another related work ëkindlmann and Durkin 1998ë uses gradient based edge detection methods to render volumes in which regions of interest are the boundaries between diæerent materials. The image-based transfer function design approach in this paper is based on a more systematic use of 3D image processing techniques ëfang et al. 1998ë, of which the approach in ëkindlmann and Durkin 1998ë is a special case.

6 6 2 Methods Interactive 3D microscopy data exploration can be achieved through a combination of interactive volume rendering and intuitive transfer function design. Technical details of this approach will be given in this section. These include a new interactive volume rendering algorithm using 2D texture mapping, and a transfer function design method based on image processing operations. 2.1 Interactive Volume Rendering by 2D Texture Mapping This algorithm applies 2D texture mapping in a shear-warp based volume rendering process to achieve interactive speed. There are two important advantages in using 2D texture mapping over other software and hardware based methods. First, 2D texture mapping is normally implemented in hardware, and therefore is faster than equivalent operations using CPU computations. Secondly, unlike 3D texture mapping hardware which is only available on selected high-end graphics workstations, hardware implemented 2D texture mapping is widely available and is usually a standard feature on most desktop workstations and personal computers. Combining 2D texture mapping and a shear-wrap factorization technique, we are able to achieve interactive volume rendering without special hardware requirements. Volume rendering using shear-wrap factorization was ærst proposed in ëcameron and Undrill 1992ë and later optimized in ëlacroute and Levoy 1994ë. Although the algorithm given in ëlacroute and Levoy 19 is one of the fastest, it still does not provide interactive rendering performance. More importantly, the algorithm carries two limitations that make it unsuitable for data exploration applications: è1è the algorithm slows down considerably when using perspective projection; è2è it requires an expensive pre-processing step for data classiæcation with every change of the transfer function. Unfortunately, both the perspective viewing and the continuous change of transfer function are crucial features in data exploration. Shear-warp algorithm is based on the shear-warp factorization of the viewing matrix: M = P æ S æ W

7 7 where P is a permutation matrix that transposes the coordinate system to allow z-axis to be the principal viewing axis, S is a shearing transformation, and W is a warping matrix that is computed by W = S,1 æ P,1 æ M. The basic steps of the shear-warp algorithm are the following: the volume data set is ærst transformed to a sheared object space by translating, scaling and resampling the slices of the volume; These slices are then composited together in a front-to-back order, which essentially projects the slices onto an intermediate image in the sheared object space; A warping operation is ænally applied to this intermediate image to generate the correct image using the warping matrix W. This process is illustrated èupper branchè in Figure 1. The main computational cost in this process is the resampling of the slices in the sheared object space and the subsequent composition of the slices. For perspective viewing, this can be particularly expensive since each slices are scaled diæerently, thus needs to be resampled diæerently. Our approach considers a slice to have two components: the image component which is the 2D image of the slice from the original data set, and the polygon component which represents the rectangular geometry of the slice. Every time the polygon is geometrically transformed èe.g. translation and scalingè, an image resampling needs to be done for the rasterization of the slice in the frame buæer. This process can, however, be accelerated by graphics hardware if the image of each slice is separately deæned as a 2D texture, and mapped to its polygon when it is drawn to the frame buæer by the graphics subsystem. Since texture mapping process involves the resampling computation èby hardwareè, this is a much faster operation than a CPU-only solution, as shown in Figure 1. As in the original shear-warp algorithm ëlacroute and Levoy 1994ë, three sets of the slices of the volume need to be deæned for the three diæerent major viewing axes. The polygon of each slice can be generated on-the-æy during rendering, but its texture image needs to be pre-deæned and stored in the system for fast texture mapping. Since both parallel and perspective viewings of polygons are handled automatically by the graphics subsystem, there is virtually no speed diæerence between parallel and perspective projections. Unlike the algorithm given in ëlacroute and Levoy 1994ë where special data structures èe.g. run-length

8 8 encodingè need to be reconstructed every time the transfer function is modiæed, the new algorithm extracts the texture images directly from the original data set, independent of the transfer functions, and therefore does not require extra pre-processing when editing the transfer functions. Finally, the wrapping step can also be conveniently carried out by 2D texture mapping. This algorithm, however, has two drawbacks. 1. Memory requirements. Since no data compression is employed for the three sets of texture images, memory requirement is large. For instance, a volume would require over 48 MB memory. Similar amount of memory is required for typical microscopy data sets of size 512 æ 512 æ 64. But since memory price has been dropping at a faster pace than other hardware components, this may not be a major concern for most users. 2. Lack of shading. 2D texture mapping can not eæciently support shading that displays more realistic surface features. Thus, our algorithm does not generate shaded images, which may result in a lose of quality for surface-rich data sets. But this does not appear to be a major problem with microscopy data sets in which surfaces are normally not well deæned due to the nature of æuorescently-labeled samples. Our experiments show that unshaded images are often more eæective for microscopy data, and shading, in many circumstances, appears to highlight noises more than the surfaces. Therefore, this algorithm is more suitable for 3D microscopy data sets. 2.2 Image-Based Transfer Function Design The need for transfer function design comes from the dynamic and often subjective visualization goals and requirements in microscopy data exploration, where the users need to interactively search and manipulate the transfer functions in the visualization process to view diæerent types of sub-structures, surfaces and frequencies with diæerent visual attributes. Thus, an intuitive and eæcient transfer function model is essential for this type of problems.

9 Transfer functions A transfer function is a function or a procedure deæned over the intensity-spatial domain of a volume data set. It computes a new intensity value for each sample point in the volume space during rendering. It can also be applied to the voxel points of the entire volume to reconstruct a new volume. The intensity values generated from a transfer function can be further mapped to color and opacity values using a color look-up table, representing a piecewise linear mapping, for each colorèopacity component. We call this step the coloring step. Since an intensity volume is essentially a 3D intensity image, a transfer function can be naturally considered as an image processing problem. In this model, a transfer function, F : V!V, is deæned as a sequence of mappings: F = f n æ f n,1 æææææf 2 æ f 1 where V is the volume data space, f i : V!Vcorrespond to a sequence of image processing procedures with adjustable parameters. This sequence and its parameter set uniquely deæne one transfer function in the transfer function space. restricted to be one of the following two types of mappings: For computational simplicity, f i is 1. Intensity table. It is an intensity-to-intensity look-up table representing a piecewise linear function over the volume's intensity æeld. 2. Neighborhood function. It is a function computed from the intensity values in a m æ m æ m neighborhood of a given voxel, where the neighborhood size, m, is an adjustable parameter of the transfer function. A median ælter ërosenfeld and Kak 1982ë, for instance, can be considered as a neighborhood function. A more typical example is the 3D spatial convolution, as a 3D linear ælter of a volume V with a m æ m æ m mask T : fèx; y; zè = m 2X i;j;k=, m 2 T ëi; j; kë æ V ëx + i; y + j; z + kë Some higher order image processing operations, such as dilationèerosion and anisotropic diæusion ëperona and Malik 1990ë, cannot be directly represented as a neighborhood func-

10 10 tion. But these operations are normally applied in some pre-computation processes for the deænitions of simpler functions Integrating transfer functions into rendering Volume rendering using this transfer function model requires the integration of image processing procedures into the rendering pipeline. This enables the interactive data exploration through transfer function manipulation. Diæerent integration approaches need to be used with diæerent types of volume rendering algorithms. These include point-based approach, volume-based approach and slice-based approach. Point-based approach. This approach can be applied when point is the basic data element processed through the rendering pipeline. Raycasting algorithms are the most typical of this kind. Essentially, every access to an intensity value during rendering will directly go through the computations of all the image processing procedures, as shown in Figure 2èaè. The main advantage of this approach is that it only computes the image processing operations on points that are actually used by the rendering algorithm, i.e. the visible points. Since the number of sample points used for rendering is normally much smaller èless than 10è in most casesè than the total number of voxels in the volume, this approach is more eæcient than applying the transfer function to the entire volume, particularly when frequent changes of the transfer functions are necessary. Applying intensity tables to points is very straightforward using color and opacity look-up tables, often in hardware èe.g. in OpenGLè. The integration of neighborhood functions are, however, more costly. A straightforward approach is to make recursive procedural calls to the neighborhood functions to dynamically compute the image processing results for individual sample points when they are accessed by the rendering algorithm. One problem with this process, however, is the potentially repeated computation with multiple neighborhood functions. Since each voxel can fall into the neighborhoods of several other voxels, and may therefore be accessed èand computedè multiple times when more than one neighborhood functions exist in a transfer function. When the number of neighborhood functions in a se-

11 11 quence is large, such overhead can be signiæcant. Fortunately, the problem can be alleviated by applying some buæering mechanism to store the intermediate results of each computed sample point for possible repeat accesses. For instance, a buæer can be used for each neighborhood function to store the results of all sample points going through this function. This way, the overhead can be partially or entirely eliminated. Volume-based approach. In this approach, the neighborhood function procedures in the transfer function are applied to the entire volume to generate a sequence of intermediate volumes, with the ænal volume passed to the coloring step, as shown in Figure 2èbè. This approach is suitable for rendering algorithms that do not have total control and access to individual voxels of the data set èor ineæcient to do soè. The most typical case is the volume rendering approach using 3D texture mapping. When the 3D texture mapping is implemented in hardware, the transfer function computation cannot be easily inserted into the texture mapping pipeline. In this case, applying the transfer function at the volume level is more practical and eæcient. Since 3D texture mapping based algorithms do not normally have early termination control ëlevoy 1990ë for avoiding computing points that no long contribute to the image, point-based approach does not really have a clear advantage in this case. The implementation of this approach is also simpler since the image processing and volume rendering are decoupled. For instance, special rendering hardware, such as the VolumePro chip by Mitsubishi may beused following the transfer function computation. Slice-based approach. This approach applies the transfer function to one slice of the volume at a time, as shown in Figure 2ècè. It is suitable for algorithms that access the volume in a slice-by-slice order. Our 2D texture mapping based shear-warp algorithm belongs to this category. Some 3D texture mapping based algorithms can also be classiæed into this category. For instance, in the algorithm given in ëcabral et al. 1994ë, images are formed by the texture mapping and blending of Z-planes, which are essentially resampled 2D slices that are perpendicular to the viewing direction. Compared to the volume-based approach, slicebased approach uses less memory and is also more eæcient and æexible when the algorithm needs to dynamically determine the sampling resolutions of the slices and the number of

12 12 slices to be sampled based on run-time conditions. I.e. only the samples that are actually used by the rendering procedure are computed by the transfer function. In practice, once a transfer function is deæned, its rendering result often needs to be examined from many diæerent angles before a change is made to the transfer function. Thus, it is important to be able to reuse the transfer function results for a sequence of renderings. This can be done naturally with our 2D texture mapping based shear-wrap algorithm. Essentially, each slice is computed by the transfer function before it is saved as a 2D texture. Thus, the rendering process only communicates with the 2D texture images generated from the slices, and each change of the transfer function requires a reloading of the 2D texture images. This process is shown in Figure 2èdè. Hardware acceleration is also possible for image processing computation. In our implementation, the ImageVision system on SGI workstations is used to compute chains of image processing operations. Since ImageVision takes advantage of the system's graphics hardware whenever possible, it is usually faster than pure software implementation. Other real-time image processing solutions are also available such as the DSP based image processing chips by Texas Instruments Image enhancement for data æltering In this image-based transfer function model, we mainly focus on two types of operations: image enhancement and boundary detection. The goal of image enhancement is to improve the quality of the 3D image volume for better visual appearance, based on user's visualization goals. I.e. it is basically a data æltering mechanism. Two types of image enhancement techniques are commonly used: point enhancement and spatial enhancement. A point enhancement operation applies some function to each intensity value, individually, to generate a new value. Since the result of a point enhancement operation only depends on the intensity value of the point to which it is applied, the corresponding intensity mapping can be represented as an intensity table in our transfer function model. The most common point enhancement operation is intensity modiæcations, in which the intensity curve of the input volume is altered in certain intensity intervals to increase or reduce the exposure of

13 13 the corresponding regions. Similar modiæcations can also be applied to the histogram curve èe.g. histogram equalizationè. Although the parameters of such operations èe.g. the intensity intervalsè can be deæned and adjusted directly by the users, they more likely will come from the output of some other image processing procedures, such as boundary detections, in a visualization process. A spatial enhancement operation derives the new intensity value of a given point from its neighborhood points, i.e. the result is neighborhood dependent. Therefore, spatial enhancement operations, can only be represented as neighborhood functions in our transfer function model. In general, spatial operations can be classiæed into smoothing and sharpening operations. Smoothing operations are primarily used to remove image noise. We sometimes also want to remove very small feature details in order to better present the larger features. One example is the Median ælter that returns the median intensity value in a mæmæm neighborhood. But more typical smoothing operators are often represented as 3D convolutions with spatial lowpass masks which ælter out high frequency image components. The mask represents a weighted average of the intensity values in a mæmæm neighborhood of each point in the volume. In addition to the mask size m, several other parameters may also be deæned to adjust the level of smoothing and blurring by manipulating the weights for the averaging. One example is the 3D Gaussian smoothing deæned by a Gaussian mask with parameters ç i ;ç 2 ;ç 3 2 è0; +1è : i2,è 2ç T ëi; j; kë =e j2 2ç k2 2ç 3 2 è Sharpening operations aim to enhance geometric features by emphasizing the high frequency components of the images. This can be achieved by applying a highpass ælter, such as the Laplacian-type ælter fèx; y; zè =gèx; y; zè,5 2 gèx; y; zè to the image volume. Another useful operation is the unsharp masking that blends the lowfrequency component and high-frequency component of an image volume, where the weights

14 14 of the linear combination are adjustable parameters, and represent the level of sharpening it generates. Again, most of these highpass ælters can be represented by convolution masks as neighborhood functions. An example is shown in Figure 3 where the structure in an Actin Filaments volume is visualized through enhancement operations Boundary detection for surface rendering Boundary detection operation ænds the surface boundary voxels to derive the appropriate transfer function or thresholds for surface rendering. Most 2D edge detection algorithms can be extended for 3D boundary detection. Many of these algorithms employ some convolution masks to compute the discrete approximations of some diæerential operators to measure the rates of changes of the intensity æeld ègradientsè, and then classify surface boundary voxels based on a magnitude thresholding of the gradient values. More sophisticated edge detection algorithms have also been developed in Computer Vision ëperona and Malik 1990ë. Iso-surface based approach. Iso-surface rendering requires pre-deæned iso-values to identify the iso-surfaces. Unfortunately, these iso-values are often not known in advance. Using the image-based approach, we can apply an edge detection operator to automatically derive these iso-values for iso-surface rendering. Note that extraction of the iso-surfaces ëlorensen and Cline 1987ë is not necessarily needed. For instance, we can deæne a transfer function through intensity modiæcation that renders only a layer of the surface voxels, deæned by some narrow intensity intervals surrounding these iso-values. To derive the iso-values, a histogram of all the boundary voxels from the boundary detection operator is ærst generated. The intensity values at which the histogram reaches local maxima can then be used as the surface iso-values. A smoothing operation èe.g. Gaussian smoothingè may need to be applied to the histogram ærst to remove noises. It should be mentioned that, with this approach, both the boundary detection and histogram analysis are pre-computations of the actual rendering process. It is, therefore, possible to apply higher order edge detection operations in this process. By setting diæerent scales of parameters

15 15 in the boundary detection process, a set of multiscale iso-values can also be pre-computed, and then used to deæne a set of multiscale transfer functions èas simple intensity tablesè for diæerent levels of surface rendering in data exploration. Dynamic boundary detection based approach. A second approach in using boundary detection for surface rendering is to directly apply a boundary detection operation to each sample point when it is accessed by the rendering algorithm. This allows the rendering algorithm to dynamically determine whether a sample point belongs to a surface boundary or not for appropriate rendering actions. With this approach, only simple boundary detection methods èe.g. convolution mask based detectorsè ought to be applied for speed reason. This approach is particularly useful for surfaces that cannot be simply deæned as iso-surfaces. One example is the photobleaching eæect in æuorescence microscopy, where the entire depth of the sample is illuminated with light that both excites and destroys æuorophores through photo-oxidation. When one attempts to collect serial optical sections of a sample volume, the images are characterized by an increase in the amount ofphotobleaching of each sequential plane, such that the same material may have diæerent intensity values in diæerent slices. In these cases, the boundaries may need to be identiæed using more sophisticated edge detection methods, and the data gradient is a more eæective measure of surfaces than the iso-value. As a very simple example, the magnitudes of gradients may be proportionally mapped to the opacity values in the opacity transfer function to emphasize high gradient regions for surface rendering eæect. A gradient thresholding may also be used to render only the high gradient voxels. In general, this approach requires the intensity mappings for transfer function deænition be represented as neighborhood functions, and is therefore more expensive than the iso-surface based approach. An example is shown in Figure 4 where the surfaces in a Golgi Complex volume are visualized through both iso-surface rendering and dynamic boundary detection. The isovalue is determined using the histogram curve of the boundary points.

16 16 3 Results Using the techniques described in this paper, we have developed an integrated system, called IVIE, for the interactive visualization and imaging of volume data sets. Although IVIE is designed for general volume data, it is particularly suitable for noisy and complex data sets, such as 3D microscopy data volumes, that require sophisticated transfer functions and interactive data exploration. The system is written in C++ and OpenGL 1.1, and is currently implemented on SGI OCTANE and O2 workstations. Since only 2D texture mapping hardware is used, it can be easily ported to PC platforms. Interactive volume rendering is achieved using the 2D texture mapping based shear-warp algorithm described in Section 2.1. The algorithm requires a 128 MB main memory to run sized volumes. Unlike the original shear-warp algorithm, this system shows no speed diæerence between parallel and perspective projections. On an SGI OCTANE workstation with 128 MB main memory and 4 MB texture memory, we are able to volume render a 256æ256æ64 data set at a 7 frames per second rate. The frame rate is reduced to 5 frames per second on an SGI O2 workstation with 128 MB main memory and 1 MB texture memory. Such frame rates allow users to interactively rotate and zoom inèout with full volume rendering resolution. For even better frame rate, IVIE also provides an adaptive rendering option that automatically adjusts the number of slices è2d texturesè to be used for rendering when the object is in motion. Since loss of resolution during motion is normally less noticeable, it is a very eæective feature for smooth animation and motion. Two image processing libraries are used in IVIE. The ærst is a software toolkit we wrote using C++ for both 2D and 3D image processing operations. The second is the ImageVision library by SGI. Since ImageVision takes advantages of some hardware features through OpenGL, it is faster than our software implementations. On the other hand, it is only available on SGI workstations, and would need to be replaced by either a software library or some hardware image processing chips if the system is ported to a PC platform. For 256 æ 256 æ 64 volumes, the transfer function computation without ImageVision takes from

17 17 6 to 170 seconds for each ælter. It is about twice as fast if ImageVision is used. Although the image processing computation using ImageVision is very fast, the majority of the time for the transfer function process is spent on building and loading the 2D texture objects rather than the image processing itself. The front-end of IVIE is a graphical user interface with a rendering window and a control panel that includes a number of control screens for controlling diæerent system functions, such asvolume rendering, 2D and 3D image processing, color and opacity map manipulation, transparency adjustment, movie making, and slice-based operations. Some sample control screens and rendering images generated by IVIE are shown in Figure 5.

18 18 4 Conclusions An interactive volume visualization system, IVIE, for 3D microscopy images is described. Two new techniques are employed in this system: interactivevolume rendering by 2D texture mapping and transfer function design using image processing operations. They form the basis of an interactive volume data exploration environment that is particularly suitable for 3D microscopy data sets. The system is intentionally designed to avoid using advanced graphics hardware features such as 3D texture mapping. Thus it can be easily implemented on PC platforms and other low-end workstations. Future work will be focused on the performance enhancement of the image processing computation and the PC implementation of the IVIE system. Currently, the speed of the image processing operations in IVIE is not yet interactive. We would like to investigate new image processing solutions èpossibly hardware enhancedè to achieve truly interactive transfer function design. Since the graphics accelerators on PCs often have very large 2D texture memory, we expect that the performance of our volume rendering algorithm on PCs will be even better.

19 19 References ëcabral et al. 1994:ë Brian Cabral, Nancy Cam, and Jim Foran. Accelerated volume rendering and tomographic reconstruction using texture mapping hardware. In Proc Symposium on Volume Visualization, pages 91í98, October ëcameron and Undrill 1992:ë G. G. Cameron and P. E. Undrill. Rendering volumetric medical images data on a simd-architecture computer. In Proc. 3rd Eurographics Workshop on Rendering, pages 135í145, May ëdunn et al. 1994:ë K. Dunn, S. Mayor, J. Meyer, and F. Maxæeld. Applications of ratio æuorescence microscopy in the study of cell physiology. FASEB J., 8:573í582, ëfang et al. 1998:ë Shiaofen Fang, Tom Biddlecome, and Mihran Tuceryan. Image-based transfer function design for data exploration in volume visualization. In Proc. IEEE Visualization'98, pages 319í326, October ëhe et al. 1996:ë T. He, L. Hong, A. Kaufman, and H. Pæster. Generation of transfer functions with stochastic search techniques. In IEEE Visualization 96, pages 227í234, Oct ëkindlmann and Durkin 1998:ë Gordon Kindlmann and James Durkin. Semi-automatic generation of transfer functions for direct volume rendering. In IEEEèACM 1998 Symposium on Volume Visualization, ëlorensen and Cline 1987:ë W. E. Lorensen and H. E. Cline. Marching cubes: A high resolution 3D surface construction algorithm. Computer Graphics, SIGGRAPH'87, 21è4è:163í169, July ëlevoy 1988:ë Marc Levoy. Display of surfaces from volume data. IEEE Computer Graphics and Application, 8è3è:29í37, May 1988.

20 20 ëlevoy 1990:ë Marc Levoy. Eæcient ray tracing of volume data. ACM Trans. on Graphics, 9è3è:245í261, July ëlacroute and Levoy 1994:ë P. Lacroute and M. Levoy. Fast volume rendering using a shearwarp factorization of the viewing transformation. SIGGRAPH'94, pages 451í458, ëmarks et al. 1997:ë J. Marks, and et al. Design galleries: Ageneral approach to setting parameters for computer graphics and animation. In SIGGRAPH '97, pages 389 í 400, ëperona and Malik 1990:ë Perona and J. Malik. Scale-space and edge detection using anisotropic diæusion. IEEE Transaction on Pattern Analysis and Machine Intelligence, 12è7è:629í639, ëparker et al. 1998:ë S. Parker, P. Shirley, Y. Livnat, C. Hansen, and P. Sloan. Interactive ray tracing for isosurface rendering. In Proc. of IEEE Visualization'98, pages 233í238, ësgi 1998:ë SGI Technical Publications. OpenGL Volumizer Programmer's Guide. SGI, ërosenfeld and Kak 1982:ë Azriel Rosenfeld and Avinash Kak. Digital Picture Processing. Academic Press, ëshaw 1995:ë P. J. Shaw. Comparison of wide-æeldèdeconvolution and confocal microscopy for 3D imaging. In Handbook of Biological Confocal Microscopy, 2nd Edition, pages 373í387, ëupson and Keeler 1988:ë Craig Upson and Michael Keeler. V-buæer: Visible volume rendering. Computer Graphics, SIGGRAPH'88, 22è4è:59í64, August ëwestover 1990:ë Lee Westover. Footprint evaluation for volume rendering. Computer Graphics, SIGGRAPH'90, 24è4è:367í376, August 1990.

21 21 Figure 1: Shear-warp algorithm: Shear-warp volume rendering by resampling and by 2D texture mapping. Figure 2: Integrating image processing into the rendering pipeline: èaè Point-based approach; èbè Volume-based approach; ècè Slice-based approach; èdè 2D texture based approach. Figure 3: A æuorenscently labeled Actin ælaments volume: èaè Volume rendering by a linear ramp transfer function; èbè Laplacian masking followed by unsharp masking with æ =3;ècè Laplacian masking followed by unsharp masking with æ = 10. Figure 4: Surface rendering of a Golgi Complex: èaè Volume rendering by a linear ramp; èbè The histogram of boundary points; ècè Iso-surface rendering with an iso-value 87, obtained from ëb"; èdè Surface rendering by dynamic boundary detection. Figure 5: Color images: IVIE rendering results of several confocal microscopy data sets, and two sample control screens

22 22 shearing & scaling projection & compositing resampled image slices original shear-warp algorithm warping shear-warp with texture mapping shearing & scaling texture images image plane draw polygons with blending & texture mapping texture mapping warping polygons

23 23 Parameter Modification Coloring & rendering Parameter Modification Coloring & rendering V0 point f1 f2 f3 point V0 f1 V1 f2 V2 f3 V3 (a) (b) Parameter Modification Coloring & rendering Parameter Modification Coloring & rendering V0 slice f1 f2 f3 slice V0 slice f1 f2 f3 Stack of 2D textures (c) (d)

24 24 (a) (b) (c)

25 25 (a) (c) 87 (b) (d)

26 26