Virtual Reality Techniques for the Visualization of Biomedical Imaging Data M. A. Shaw, w. B. Spiliman Jr., K. E. Meissnera, Gabbardc athe Optical Sciences and Engineering Research Center, Virginia Polytechnic Institute and State University; of Physics, cresearch Graduate Studies, Virginia Polytechnic Institute and State University Abstract The Optical Sciences & Engineering Research Center (OSER) at Virginia Polytechnic and State University investigates advanced laser surgery optics, biocompatible material for implants, and diagnostic patches and other diagnostic and drug delivery tools. The Center employs optics to provide new biological research tools for visualization, measurement, analysis and manipulation. The Center' s Research into Multispectral Medical Analysis and Visualization techniques will allow human and veterinary medical professionals to diagnose various conditions of the body in much the same way that satellite information is used to study earth resources. Each pixel in the image has an associated spectra. Advanced image analysis techniques are combined with cross-correlation of the spectra with signatures of known conditions, allowing automated diagnostic assistance to physicians. The analysis and visualization system consists of five components: data acquisition, data storage, data standardization, data analysis, and data visualization. OSER research efforts will be directed toward investigations of these system components as an integrated tool for next generation medical diagnosis. OSER will research critical data quality and data storage issues, mult-spectral sensor technologies, data analysis techniques, and diagnostic visualization systems including the VT-CAVE,(www.cave.vt.edu). The VT-CAVE is Virginia Tech's configuration of Fakespace Systems, Inc Virtual Reality system. Introduction The Optical Sciences and Engineering Research Center (OSER) was recently formed at Virginia Tech. The center is a collaborative effort between Virginia Tech and the Carilion Biomedical Institute. OSER is specifically tasked with conducting research and engineering activities involving optics and other disciplines to create knowledge and technology to benefit the medical, biomedical and veterinary fields, while supporting the practical goals of improving services and reducing the costs of health care. Multispectral medical analysis and visualization can allow human and veterinary medical professionals to diagnose various conditions of the body in much the same way that information from satellites can be studied to yield information about the earth's surface (vegetation, pollution, and mineral resources, for example). Each pixel in the image has an associated spectra. Advanced image analysis techniques are combined with crosscorrelation of the spectra with signatures of known conditions, allowing automated diagnostic assistance to physicians. 24 Biomarkers and Biological Spectral Imaging, Gregory H. Bearman, Darryl J. Bornhop, Richard M. Levenson, Editors, Proceedings of SPIE Vol. 4259 (2001) 2001 SPIE 1605-7422/01/$15.00
Multispectral Medical Vizualization Advances in sensor technology using multispectral and hyperspectral data acquistion systems can be combined with immersive data visualization techniques to produce enhanced reconstructions of a patient's condition for diagnostic purposes. These emerging systems will require developments in sensors, data standards, data management, computational data analysis, and improved data visualization systems. Advances in multispectral and hyperspectral technology produce data rich images. These images can be combined using computational techniques to produce information rich products. Data compression and transmittal standards and processes for these images must be developed to ensure quality is maintained. Furthermore, the computational algorithms combined with immersive stereoscopic visualization techniques can result in enhanced diagnostic capabilities. A rigorous information system is required to support this research and development effort. The system must be modifiable, flexible, robust, scalable, maintainable, and have a set of supporting process in place to ensure data integrity, quality control, and availability. The critical challenge is the development of techniques that will provide the clinician an enhanced diagnostic space. This will require development of new methods for information portrayal derived from new sensor technologies. Display of enhanced three dimensional image elements within an immersive environment and interacive analysis techniques must be developed. Finally, this new diagnostic space must provide access to all available patient and other critical information within the work area to provide a complete decision support capability. Medical Visualization System Components The goal of a multispectral data visualization system is to provide enhanced diagnosis capabilities for use by the medical practitioner. The system consists of five components: 1. multispectral data acquisition; 2. data management; 3. data reduction; 4. data analysis; and 5. stereoscopic visualization, (Figure 1.). The data acquisition and visualization systems will provide enhanced capabilities for portraying multispectral data abstractions within a natural three-dimensional stereoscopic immersive display system. Multispectral Data Acquisition Multispectral imaging systems acquire full spectral data at each pixel of an image. Thus, a three-dimensional data cube is built with the axes being x, y and wavelength (X). The multispectral data acquisition system used at OSER was custom built by OKSI (Torrance, CA). It consists of a TE-cooled, blue enhanced Silicon CCD array, a liquid crystal tunable filter (LCTF), camera optics, and a laptop computer for control and storage. Either a visible LCTF (400nm 720nm) or a near-infrared LCTF (600nm lo5onm) may be fitted on the camera system. The image acquisition time depends on the integration time at each wavelength band, but is generally on the order of one minute. An image cube is 512 x 512 x 33 (39) for the visible (NIR) and occupies approximately 17MB (20MB). Data Management The massive amounts of data generated by the enhanced sensor systems must be managed and stored in a manner that preserves the quality and unique identifying characteristics of each data set. The meta data describing the sensor data must include machine calibration records, patient identification, patient condition at the time of the measurement, a complete Proc. SPIE Vol. 4259 25
Figure 1. Component Diagram of a multispectral medical data acquisition, analysis, and Visualization system. record of data reduction and/or computational modification to the original data. This information is critical to ensuring data quality is maintained by preserving the chain of events and processes that result in the analytical or visualization product being evaluated by the clinician. The OSER system will support the research effort and therefore data provenance is critical when evaluating data anomalies and determination of the efficacy of computational algorithms. The data maintained by the OSER data management system will be invaluable for evaluating new diagnostic tolls as techniques are developed by the research team. Also included in the data structure will be a complete set of patient records. This information will be available to the analyst and diagnostician during computational analysis and also within the virtual environment to provide an effective and complete virtual diagnostic space. The availability of this information is critical to the proper analysis of factors that may contribute to the variances in the sensing system. Changes in tissue response due to the physiological state of the patient due must be available to the development team. Data Reduction Data reduction transforms raw multispectral sensor derived data into more useful forms. The techniques used during transformation will be preserved along with the resulting data set to ensure data provenance is preserved for quality control. The data reduction products will include data transformation into industry standard formats for data analysis, image visualization, and three-dimension virtual reconstruction. 26 Proc. SPIE Vol. 4259
Data Analysis The data analysis component will consist of integrated commercial-off-the-shelf (COTS), open-source, and custom software to be used for computational analysis. The analytical component will be tightly coupled with the visualization system to allow interactive stereoscopic visualization. This capability will be available both at desktop workstation and with the VT-CAVE immersive environment. The data analysis toolkit will include both COTS and custom computer applications. Appropriate software applications used by the remote sensing industry will be applied (and customized as necessary) to support the multispectral diagnostic development effort. It is expected that research into the efficacy of applying these reliable remote sensing techniques to medical multispectral data is a promising research area. Visualization Data visualization is the science of turning sets of complex data into high-density visual information and understanding. Current and emerging technologies allow the generation of virtual environments (VEs) that provide access to traditional data visualization analysis techniques and the ability to combine these capabilities with scaled digital models, video, and other multimedia within an interactive immersive, stereographic environment. Advanced visualization has become a mainstream tool in applications where complex data sets need to be studied and manipulated. The ability to provide human-scale display, immersion, visual databases, spatial integration, and collaboration solutions will have a significant impact on clinical and diagnostic procedures. The technology provides immediate sensation of the spatial relationships between data. Data can be portrayed using techniques that can either abstract complex data into recognizable patterns or portray information in a natural and therefore more understandable context. The use of VEs for data fusion is an area of active research. The technology currently provides capabilities for:. interactive work environments with access to traditional data analysis techniques, S any combination of spatially related data (or abstractions of the data), computational products, or data classifications,. 3D data models from other sources,. 3D models produced by voxelation of image slices. video, S images,. audio and 3D audio,. scale models of environments, structures, sensors, and devices,. interactivity at any scale or from any viewpoint,. textual labels, visual, and audio information that would help investigate, understand, and communicate a medical investigation,. interactive devices that can be developed specifically to aid in the dissection of discrete data images, interactive tools to manipulate and navigate the VE scene graph, and, interactive collaborative work sessions with remote sites. The OSER will be using desktop workstations and Virginia Tech's VT-CAVE immersive system for medical visualization and diagnostic analysis. The CAVE(tm) is a multi-person, room-sized, high-resolution, 3D video and audio environment. In the current configuration, graphics are rear projected in stereo onto three walls and the floor, and viewed with stereo Proc. SPIE Vol. 4259 27
glasses. As a viewer wearing a position sensor moves within its display boundaries, the correct perspective and stereo projections of the environment are updated by a supercomputer, and the images move with and surround the viewer. Hence stereo projections create 3D images that appear to have a presence both inside and outside the projection-room continuously. To the viewer with stereo glasses the projection screens become transparent and the 3D image space appears to extend to infinity Specifically, the CAVE(tm) is a theater loxlox9 feet, made up of three rear-projection screens for the front, right and left walls and a down-projection screen for the floor. Electrohome Marquis 8000 projectors throw full-color workstation fields (1024x768 stereo) at 96 Hz onto the screens, giving approximately 2,000 linear pixel resolution to the surrounding composite image. Computer-controlled audio provides a sonification capability to multiple speakers. A user's head and hand are tracked with Ascension tethered electro magnetic sensors. Stereographics' LCD stereo shutter glasses are used to separate the alternate fields going to the eyes. A Silicon Graphics Power Onyx with three Infinite Reality Engines is used to create the imagery that is projected onto the walls and floor. (http://www.sv.vt.edu/future/vt-cave/whatis/) Conclusion The goal of a multispectral data visualization system is to provide enhanced diagnosis capabilities for use by the medical practitioner. The system consists of five components: 1. multispectral data acquisition; 2. data management; 3. data reduction; 4. data analysis; and 5. stereoscopic visualization. The full spectra imaging system is combined with the data analysis and interactive visualization to provide means for evaluating sensor response, computational alogorithms, and efficacy of the portrayal techniques as a diagnostic tool. The data management component provides a platform for storage of raw and computational data and the resulting visualization products. This component also provides the patient information necessary to a comprehensive diagnostic space. Finally the data reduction component provides the required procedures for standardizing and reducing the data acquired from the sensor systems. The stereoscopic workstations and the VT-CAVE, a multi-person, room-sized, high-resolution, 3D video and audio environment, will be used for medical data visualizations and research into diagnostic procedures. In conjuction with this data visualization work, we investigate an intriguing method for analyzing hyperspectral data in talk 4259-10, "Cellular Automata for the Analysis of Biomedical Hyperspectral Images", later in this session. Here, cellular automata are used to rapidly scan hyperspectral images and quantify the extent of conditions of medical interest. This technique can lead to a large reduction in the computational time spent analyzing large hyperspectral images. 28 Proc. SPIE Vol. 4259