Supervised Segmentation of Visible Human Data with Analogies James B. Lackey and Michael D. Colagrosso Department of Mathematical and Computer Sciences Colorado School of Mines Golden, CO 80401 {jlackey,mcolagro}@mines.edu Phone: 303-384-2465 Fax: 303-273-3875 Abstract We present a new application of the Analogies algorithm to be used for image segmentation. Our approach requires supervised training data, so we apply it to the domain of labeling human anatomical data. In the Visible Human Project, expert anatomists are overwhelmed with high-resolution images to analyze. We propose that the anatomist can work in conjunction with our approach, letting the machine segment 80% of the images, and requiring that the expert segment only every fifth image. Keywords: image analogies, image segmentation, visible human I. INTRODUCTION We consider image segmentation in a domain that facilitates supervised learning. Unlike unsupervised learning methods, such as spectral clustering [2], we are trying to directly emulate the segmentation of an expert anatomist. In the medical industry, far more data on the human body is being collected than can be properly analyzed. There is no reliable way to quickly segment large volumes of data, and even when this data is segmented, it may stack to form a poor 3D model. If we can partially automate segmentation, then medical data can be analyzed faster and at smaller cost. Furthermore, we can impose constraints on the segmentation process that will lead to the creation better 3D models of human structures. II. THE VISIBLE HUMAN PROJECT The Visible Human Project [3] has collected large amounts of data to be segmented. The anatomical images of the Visible Human Male were created with vertical and planar resolution of 1 mm, yielding approximately 1800 slices to be segmented. Each image contains dozens of structures which need to be segmented. Some, such as the colon or appendix are relatively large. But, others, such as cross sections of arteries or even blood vessels are much smaller. Anatomists segment slices one structure at a time, so segmentation of all the organs in a set of slices requires several passes. Currently, a human segments the images by hand with the aid of Photoshop-like tools. This is very time consuming. One goal of the visible human project is to build instructive 3D models of human organs, such as the kidneys, heart, and lungs. Taking the kidney as an example, the model is formed by stacking all the slices that contain the kidney and combining all the points in the slices that are members of the kidney segmentation. One problem with human segmentations they often do not form smooth 3D models. A method which imposes a curvature restriction on the structures within the model would perform much better. Segmentation Fig. 1. Sample visible human image and its human segmentation. Solid colors in the segmentation represent bone, muscle, fat, etc. III. PREVIOUS SEGMENTATION METHODS In the past, deformative models have had the most success automatically segmenting data. [4] These start with a human segmented image and deform the boundaries as proximate slices are presented. Deformative models perform well for contingous structures. These models attempt to identify boundaries and then segment on the basis of these boundaries. They typically do not incorporate knowledge about textures or textons. However, these deformative models are limited. They can only account for changes in shape of structures; they cannot correctly segment structures that were not present in the original slice. IV. IMAGE ANALOGIES Analogies (IA) [1] gives an algorithm to solve the problem A : A :: B : B where B is unknown. For English words, A relates to A in the same way B relates to B. Analogies (IA) can be used to map the texture of one image
onto another image, synthesize analogous texture, or compute texture by numbers. The synthesis of texture by numbers is most pertinent to our work. In texture by numbers, IA is given segmented images A and B, and an unsegmented image A. IA works by constructing Gaussian pyramid representations 1 4 resolution representations of A, A and B. For each pixel in B, a corresponding region in A is selected using the Gaussian pyramids. For that pixel, B is assigned the same value as A. This 1 4 resolution image is then sampled to produce a 1 2 resolution Gaussian pyramid. The remainder of the pyramid is computed. Finally, a full resolution pyramid is constructed using the 1 2 resolution pyramid and additional computation. V. APPLICATION Up to now, IA has been applied to do texture by numbers. That is, given a segmented image, IA can compute the unsegmented image. We want to do the opposite, compute a segmented image given an unsegmented image. That is, we want to use IA as a texture recognizer. We have a set of consecutive Visible Human slices: {m 0, m 1,..., m n } and we have a human segmented image s 0. The most naive application of IA would be to compute the analogies: m 0 : s 0 :: m 1 : s 1 m 0 : s 0 :: m 2 : s 2... m 0 : s 0 :: m n : s n However, this gives poor results. As the image being segmented gets farther and farther from m 0, accuracy drops off. Performance can be improved significantly by progressively computing analogies. That is, we compute m 0 : s 0 :: m 1 : s 1 m 1 : s 1 :: m 2 : s 2... m n 1 : s n 1 :: m n : s n In the results below, we use the anatomist segmentations as our basis for estimating performance in terms of accuracy, precision, and recall. Additionally, we desire a method that will provide segmentations that provide smoothness and therefore yield useful 3D models. Discrepancies between layers generated by IA will typically be small, so the model will be smooth. VI. RESULTS We present three sets of results. First, the results of IA segmented all structes in Visible Human slices. Second, we present segmentations of muscle tissue, and finally segmenations of the colon. A. Full Segmentation The naive method of segmentation consistently performs better than the progressive method on accuracy, precision and recall. However, these results are misleading. The progressively segmented images are more smooth within the slice and form a more smooth 3D model when stacked. For this reason, we need a different measure of segmentation performance. The original goal is to build smooth 3D models. Accuracy, precision and recall say nothing about smoothness, so we must devise our own smoothness metric. Say we want a normalized measure of smoothness. We define the smoothness of a pixel at coordinates (x, y, z) to be 1 if the pixels (x, y, z), (x, y, z 1) and (x, y, z + 1) have 1 the same value, 2 if (x, y, z) or (x, y, z 1) has the same value as (x, y, z) and 0 otherwise. We can then take the average smoothness over a segmentation by taking the average of smoothness across all pixels in each slice of the segmentation. TABLE I AVERAGE MEASURES OF ACCURACY ±SD FOR 9 SIMULATIONS Naive Progressive 0.450± 0.025 0.504± 0.009 0.438± 0.022 0.466± 0.025 0.434± 0.041 0.457± 0.025 0.389± 0.025 0.432± 0.025 The only significant errors in the segmentation are in images 2,3 and 4 of Figure 2. The mislabeled region is the psoas major right that is labeled as psoas major left. The psoas major left and right are muscle tissue on either side of the spine. They are symetric. Our experts segmented psoas major left, erector spinae left and quadratus lumborum left as one texture, but decided their symmetric counterparts would be a different texture. Clearly this choice is somewhat arbitrary, as there is no functional difference in the textures representing the muscle tissue. For this reason, it is understandable, and perhaps even acceptable, that IA was unable to distinguish between these two textures. B. Muscle Tissue Segmentation The full segmentation images in the previous section have some structures labeled in a way that is not conducive to learning. For example, in Figure 2, there are several muscle textures in the image, but they are labeled differently in the expert segmentation depending on whether they are on the left or right side. The IA algorithms cannot learn this anatomical distinction; the Progressive IA algorithm, for example, labels the left muscle texture the same as the right in s 2, 3, and 4 in Figure 2, resulting in large errors. Based on this result, we focus on correctly labeling one single texture, which is a similar to the anatomist s method of labeling one structure and making several passes. Once we
Expert Naive IA Progressive IA Prog. IA Error Fig. 2. Results of Full Segmentation. The Progressive IA approach labels most of the textures correctly, but labels the left muscle texture the same as the right, resulting in the large black regions in the Error column. have labeled all the muscle textures in an image, an anatomist can further subdivide the them based on what muscle groups they belong to. In Figure 3, we created the expert image by labeling any type of muscle white, and everything else black. As expected, the resulting errors are much smaller in this limited task. C. Colon Segmentation The labels of muscle texture in Figure 3 are especially sensitive to the small amounts of fat that are intermingled. In fact, most of the errors in the Progressive IA approach are in the interior of the large muscle group, which would be tedious for an anatomist to correct. Instead, we desire a texture that our method can label accurately and require the least postprocessing by the anatomist, even if it is on a more-limited scope than the full segmentation. To this end, we applied our method to the colon structure in the same manner as the muscle above. The results in Figure 4 show an accurate segmentation, and the errors are on the boundaries of the region, not the interior. TABLE II AVERAGE M EASURES OF ACCURACY FOR 5 NAIVE C OLON S EGMENTATION ±SD Accuracy 0.986±0.001 0.977±0.003 0.967±0.010 0.942±0.035 Precision 0.955±0.007 0.903±0.012 0.911±0.009 0.838±0.149 Recall 0.956±0.007 0.954±0.004 0.869±0.065 0.911±0.017 VII. C ONCLUSIONS AND F UTURE W ORK By progressively segmenting, we can produce reasonably accurate segmentation four slices out. In short, we can reduce the anatomist s work by 80%. One limitation of the human and machine learning segmentations is that a pixel can be a part of only one segment. Thin structures are sensitive the consequences, and discontinuities can arise. For the human segmentation, this is a consequence of the tools used to decompose the image, but the machine learning approach has no such limitation.
Expert Naive IA Progressive IA Prog. IA Error Fig. 3. Results of Muscle Tissue Segmentation. By labeling all the muscle textures white and everything else black, both approaches perform better, but still make several one-pixel errors that would require review by an anatomist. TABLE III AVERAGE M EASURES OF ACCURACY FOR 5 P ROGRESSIVE C OLON S EGMENTATION ±SD Accuracy 0.990±0.0001 0.985±0.0004 0.978±0.0002 0.972±0.0011 Precision 0.970±0.001 0.949±0.001 0.924±0.003 0.899±0.008 Recall 0.967±0.001 0.964±0.004 0.940±0.002 0.923±0.002 In the future, we will generalize IA to higher dimensionality. The obvious way to do this is to compute a Gaussian hyperpyramid over some set of image {m0, m1,..., mn } for which we have segmented images {s0, s1,..., sn }. After computing the pyramid we could compute matches for the images {sn+1, sn+2,..., sm } One advantage of this methodology is that it would impose smoothness on the image. Another is that it incorporates information about consecutive layers. For instance, in the 2D case, the machine might know that the backbone occurs next to other pixels of backbone, but it would not know that long sections of backbone occur vertical. The Gaussian hyperpyramid would incorporate this knowledge. We believe we can further improve performance by using a Canny Filter as a post-processor. Given our machine segmented images {sn, sn+1,..., sm } and a set of closed regions defined by the Canny filter, we can define a new segmentation with the boundaries from the Canny filter. The regions defined by those boundaries will be segmented according to whatever texture was dominant within the segmentation {sn, sn+1,..., sm }. This gives a new segmentation. ACKNOWLEDGMENTS Thanks to Elizabeth Prince from the CU Health Sciences Center for insight into the Visible Human data and thoughtful explanations. R EFERENCES [1] Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, David H. Salesin. Analogies. Proc. SIGGRAPH, 2001.
Expert Naive IA Progressive IA Prog. IA Error Fig. 4. Results of Colon Segmentation. In this simplified texture, the Progressive IA approach makes errors only on the boundaries of the texture. [2] A. Ng, M. Jordan and Y. Weiss. On spectral clustering: Analysis and an algorithm. Advances in Neural Information Processing Systems 14, 2001. [3] http://www.nlm.nih.gov/pubs/factsheets/visible human.html [4] A. Singh, L. von Kurowski, and M. Y. Chiu. Cardiac MR Segmentation Using Deformable Models. Proc. SPIE Medical Imaging, New Port Beach, 1993.