Toward Real-Time Biopsy Image Analysis and Cell Segmentation

Transcription

1 Toward Real-Time Biopsy Image Analysis and Cell Segmentation Luis Hernandez, Paula Gothreaux, Liwen Shih Computer Engineering, University of Houston - Clear Lake (UHCL) Gerald Campbell Neuropathology, University of Texas Medical Branch Galveston (UTMB) Abstract This Digital Signal Processing (DSP) study is aimed at real-time capture and analysis of pathological brain images to improve accuracy and efficiency. Simply analyzing cell density statistics and average cell nucleus diameters of a slide image is shown to be useful to determine the abnormality of brain sample. Numerous biopsy samples of various types obtained around the world daily are sent for screening and diagnosis to enable proper treatment, often while patients may be painfully suffering the symptoms for days to weeks anxiously awaiting the biopsy results. In general, pathological image analysis using a computer-based application could demonstrate great precision and efficiency for screening large quantities of cells on one or numerous sample slides, as opposed to the tedious and error-prone human eye counting and measuring hundreds to thousands of cells in one sample slide under a microscope. As a high-level, interactive environment for data visualization/analysis/computation, MATLAB is currently utilized to perform automatic image analysis and segmentation of brain cells on a computer. By comparing cell concentration and cell nucleus sizes between cancerous and normal image groups, MATLAB can be programmed to distinguish normal brain cells from questionable ones. Currently, MATLAB image analysis works on captured/digitized slide images and takes a minute per image to automatically pre-screen abnormalities that require further human expert analysis. With future real-time/parallel/machine-intelligent improvements, we hope that DSP can help physicians, pathologists, and patients everywhere to get immediate diagnosis for timely/effective treatment, and can show accuracy within acceptable levels that are comparable to human pathologists in dealing with cell-overlapping and noncell objects existing in slide images. Keyword: Pathological Image Analysis, Morphology, Cell Density, Nucleus Diameter, Real-Time DSP Acknowledgement: Sample brain cell slide images were provided by UTMB-Galveston. 1. Computer-Based Pathology One of the biggest challenges faced in image analysis at the microscopic cellular level today is the randomness of the cells and the uniqueness of each cell, as well as the presence of other objects such as blood vessels, nerves, etc. The presence of these other objects within typical sample slides encumbers the sample slide image analysis. The presence of abnormal cells complicates the analysis even further. The purpose of this study was to evaluate and design a computer-based tool for screening pathological slide samples, providing slide specific statistics to aid in determining normality (or abnormality) of the sample. Utilizing previous research, Image Analysis and Segmentation of Brain Cells by Luis Hernandez [1], as a starting point, we expanded the philosophy to optimize the analysis methodology. The goal was to enhance the accuracy of statistical results. In addition, a Graphical User Interface (GUI) was designed to simplify the usability of the tool (e.g. a selectable image listing to expedite analyses of multiple samples). In the previous research, the method chosen to perform image segmentation was grayscale thresholding. The threshold methodology is performed by obtaining a digital picture of a slide and then by converting the picture to grayscale (divided into 256 shades of gray) for computation efficiency. Since the areas of interest are the cells nuclei, which appear darker with a circular geometry, the grayscale threshold reference is chosen to focus on these features. Once the distinction between nuclei and other objects is made, the image is filtered to highlight cell nuclei. At this point, we were left with a cell segmentation issue for cell nuclei that were touching/overlapping. They appear as one cell to the computer-based tool, which distorts the statistical data. Several enhancement techniques were attempted, such as functions like strel and morphology techniques like tophat and watershed, which will be discussed in detail later. The issue of statistical data accuracy was resolved and the optimized MATLAB script was combined with the GUI user interface to form the final product [2].

2 2. Counting and Measuring Cells on a Slide The initial MATLAB script consisted of reading an image from the computer hard drive into memory using MATLAB s function imread. Once the image was in memory, it was converted to grayscale for faster processing using the function rgb2gray. The grayscale image, in a range from 0 to 255 shades, can greatly reduce computation complexity of the original color image and can also eliminate possible complications caused by the stain/lighting used when the picture was taken. In order for the image conversion to take place, the original image has to be in.jpg format. Care must be taken when imaging the slide samples so that the pictures are not established in.gif format. The properties of.gif files are not recognized by the rgb2gray function and the script would cease operation at that point. To view the density distribution of the different shades of gray the function imhist is used to create a histogram. Typically, each cell has one nucleus, and cells with abnormally bigger nuclei are typically bigger than normal cells. For cell counting and relative cell size measuring, the cells nuclei are the items of interest in each image, because the nuclei appear darker and clearer than the rest of the objects in the picture. To simplify cell segmentation, the function imhmax is used to filter out the light areas and create a uniform background with the histogram analysis. As a result, the cells nuclei are highlighted, as can be seen in Figure 1. Figure 1. Original gray and filtered images Now that the light areas have been eliminated and the cells nuclei are highlighted, another filter is applied to eliminate the lighter areas of the image. The function imopen is used with the strel structuring function to create a binary image of the region of interest. The strel function uses the syntax var = strel (shape, parameters). In this case, we use the disk shape, parameter three for the radius of structuring element. Figure 2 shows the disk-structuring element with a radius of three. The purpose of the imopen function is to have only elements that fit the shape of the strel function segmented into the binary image. The binary image has all bits that fit the structuring element turned on, with the rest turned off. Figure 2. Disk structuring element with R = 3 The cleaned cell nuclei result of the imopen function applied to the filtered image and applied with the strel function is shown in Figure 3. Figure 3. Cleaned binary image From Figure 3, it can be seen that not all the cell nuclei are round and some areas that are not parts of cell nuclei appear in the image. To filter the extra parts from the image, the strel function is used again, but this time with the line structuring element applied in the vertical and horizontal direction in conjunction with the imdilate function. Another function, used after the imdilate is used to remove the parts that are not part of the cell nuclei, the imfill function is used with the holes parameters to fill in any nonuniformities inside the cell nuclei and make the inside of the cell nuclei uniform in the binary image. At this time, all the cell nuclei are segmented from the rest of the image, but if we are to obtain statistics of the cell nucleus sizes, the cell nuclei touching the border are not complete cell nuclei and will distort the statistics. The function imclearborder eliminates the cell nuclei touching the border from the image. The number of cell nuclei touching the borders was so small in comparison to the difference in cell numbers between normal and abnormal cells in the brain cell samples we tested that the partial cell nuclei on the borders are negligible. Alternatively, after the average nucleus size is calculated later excluding the incomplete border

3 nuclei, it is possible to estimate the equivalent total number of border cell nuclei by adding all partial cell nucleus areas to be divided by the average size, if the border cell count is needed in analyzing different types of tissue sample. Figure 4 shows the filtered and dilated image with holes filled and the cell nuclei touching the border removed. This is not entirely accurate since the cells that are touching are treated as one element. Figure 4. Border elements removed When the statistics were performed on this picture, the area of the cells that are touching was twice as large as the single cells. Further work was done to the script to segment the overlapping cells. One method that was researched is to shrink the cells in the binary image until they separate and then enlarge them back until they touch creating a mask that can be applied to the original image to segment the individual cells. To visualize the accuracy of the script, the binary image shown in Figure 4 is outlined and superimposed on the original image. Figure 5 shows visually how the script outlined the cell nuclei. The bwlabel function is used to label the connected components of the binary image and this data can be used to get a count of the cell nuclei using the numobjects function and the area of the connected components using the regionprops function. Figure 5. Original with outline of cell nuclei As can be seen from Figure 5 above, the problem with this un-optimized script is that the overlapping cells are counted as one cell. One of our challenges was to determine a way to segment the touching cells in order to be able to produce more accurate statistics. The data obtained with un-optimized script from this picture was 15 objects with an average size of 220 pixels. From the distribution chart generated, we get the following statistics as in Table 1. Table 1. Area and distribution of cell nuclei Number of Objects Area in Pixels Accurate Overlapped-Cell Segmentation Several functions built into MATLAB were used to attempt optimization of the cell segmentation process (to increase accuracy). Specifically, the image segmentation functions (already in the code) were tweaked with new variables and rearranged multiple times. New functions were also inserted appropriately (i.e. watershed techniques) to attempt this optimization. However, no consistency was found that would provide acceptable results. After some research, the SDC Morphological Toolbox for MATLAB was found. The toolbox works with MATLAB, and it enhances some of its functions dealing with image analysis/segmentation. An evaluation copy was downloaded and integrated into our project to evaluate how it could improve the previous script. There was significant improvement. The new script starts similar to the previous one by reading the original color image into memory. The original image is converted to grayscale using the rgb2gray function for simplicity, as it was done before. From the histogram, it is determined that setting a grayscale threshold of 200, removed most background noises, leaving the cell nuclei for further analysis. (The threshold value can be tuned based on image brightness or for tissue samples in another application.) To remove any particles that are not cells and fill in any pores within a cell, the mmareaopen is used from the morphology toolbox. Figure 6 shows the original picture converted to grayscale and the threshold applied turning the image to a binary format to display the nuclei and background areas. As can be seen, there is noise in the image that needs to be removed. To remove the noise, a function to smooth the edges of the cells and

4 consider only objects that fit an octagonal geometry of a specified size is applied. Figure 8. Markers applied to peaks and watershed applied Figure 6. Original picture and mmareaopen function Figure 7 shows how the noise is filtered from the image and the cell nuclei appear to have a smooth geometric shape. Up to this point, not much has changed from what the original script had done, but the next function applied is what makes the segmentation of the touching cell nuclei possible. The areas where the cells are located are analyzed by dilating the region maximum of the distance transforms for each cell nucleus. The cell nuclei are then displayed as topographical images as can be seen also in Figure 7 below. By looking at both pictures in Figure 8, it can be noted that if a peak can be identified, the cell can be segmented. There is one cluster of two cells that were not segmented, but from the figure in the left, it can be seen that only one peak was identified for the cluster and by looking at the original picture, one of the cells in the cluster had a light center. More research needs to be performed to determine if light areas in a cell prevent them from being identified and segmented. The watershed division lines are applied to the segmented image to make a clear separation of the touching cell nuclei. Figure 9 shows the watershed lines superimposed to the segmented image and in the other picture, the lines are removed leaving segmented cell nuclei only. Figure 7. Smoothed edges and topographical surface At this point, the image appears to be a series of mountains with each mountain peak representing an individual cell nucleus. From Figure 7, it can be seen how three cell clusters of two touching cells now appear to be two touching mountains. To visualize the individual mountain peaks, markers are applied to every peak as can be seen in Figure 8. The watershed function starts at the peaks expands in every direction until it reaches an edge from another peak or the edge of the picture. Figure 8 shows the segmentation of every individual mountain including the ones that were touching. Figure 9. Watershed lines applied to segmented image and lines removed Segmenting the cells in the manner described above not only improves the accuracy of the cell count, but also the accuracy of the cell nucleus area statistics. In order to make statistics with regards to cell nucleus area even more accurate, we remove any cell nuclei that are touching the border, since most of those cells are more likely to be incomplete cells. Again, after the average nucleus size is calculated, excluding the incomplete border nuclei, it is possible to estimate the equivalent total number of border cell nuclei by adding all partial cell nucleus areas to be divided by the average size, if the border cell count is needed in

5 analyzing different types of tissue sample. It is also possible to double count the projected overlapping nuclei area (intersection) to improve the accuracy of the cell nucleus area statistics. After the cell distribution graph is displayed, the statistics for cell counts and average area of the cell nuclei are calculated and displayed. For this example, the number of cells found was optimized up to 20 and the average cell nucleus size was down to pixels. 4. GUI for Pathology Image Analysis Figure 10. Border cell nuclei removed and final segmented image Figure 10 above shows a cell count of 20 cells, that is five more cells than what the original script found. To be able to see which cells are identified and which are missed, an outline of the segmented cell nuclei are superimposed over original grayscale image, as shown in Figure 10. For ease in visualizing the accuracy of the script, the picture used in this example is a 20-cell subsection of a full slide that has over 1000 cells. Once the script was optimized and acceptable results were obtained, the script was used to obtain statistics on several samples containing normal and abnormal cells. We will discuss the statistics obtained later. The last step of the script was to display a graph showing number of cells vs. cell nucleus area to see the cell distribution in a graphical format. Figure 11 below shows a cell distribution graph for the analysis described above. From the graph, it can be seen that most of the cell nuclei have an area between 125 and 175 pixels, with two cell nuclei being larger than the rest, indicating that they didn t segment properly, and one being smaller than the rest, indicating the presence of a very abnormal cells or improper segmentation. Figure 11. Graph of cell distribution At the time the script was being optimized, it was realized that a GUI-based, user interface was needed to be able to easily change the image file to be analyzed. The image file to be analyzed was hard coded into the script along with the path, and every time a new image was to be analyzed, the script would have to be edited with the new name/path of the file. To solve this problem, it was decided to have all image files in the directory c:\images. The script looks in that directory by default, and the GUI displays a listing of all the image files in the default directory. The user can simply select the file to be analyzed. Once an image file is chosen, the analyze button is selected to start the analysis of that image file. The script mentioned above is executed, performing all image segmentation calculations, and displays the results in the GUI window. The number of cells counted is displayed in a text box, along with the average cell nucleus size in pixels. The user has the ability to compare the original image with the segmented image to be able to see the accuracy of the analysis by visual inspection of the two files. An important part of the analysis is to be able to see the cell distribution in a graphical format, so the final analysis graph was added to the GUI display window. A template for the user interface was created using the GUIDE MATLAB function. The template has areas with axes to display the pictures and graphs. A list box was added to display a listing of all the images in the default, image directory. Two text boxes were added to the template to display the numerical results of the script, and a push button was included to give the user the ability to start the analysis of a new image, at will. Figure 12 shows the end result after a file is chosen and the analyze button is selected. The properties for the text boxes are initialized to 0 at the execution of the script and are updated at the conclusion of the each script execution. Editing the property tab of each object in the template sets the properties and initial values for each object in the template. Here, the value for the number of cells and average size were reset to zero and the list box was set to display the path to the default directory c:\images.

6 Figure 13 shows a clear separation between the normal and abnormal cells. The normal slide images have cell counts between 100 and 300 cells per image, while the abnormal ones vary from 400 to 800 cells. Keep in mind that these statistics were taken with the un-optimized script and touching cells were counted as one, so the statistics shown here count fewer than actual number of cells with inflated cell nucleus sizes. Figure 12. GUI window design All of the intermediate steps described before are not displayed while using this interface. Only the original and end result segmentation image are displayed along with the statistics and cell distribution. To run an analysis on a different image, it is just a matter of selecting a different image file from the list box and selecting the analyze button. If more files need to be analyzed, those images must be added to the c:\images directory and they will appear in the list box. Statistics were obtained from all the images of normal and abnormal cells to try to determine a clear separation between the slide samples. 5. Statistics - Cell Count and Nucleus Size A series of normal and abnormal slide samples were obtained and analyzed with the original script. The statistics obtained from the original cell segmentation script were graphed to display the number of cells for particular types of cells. For this study, the normal slide samples had normal white matter, normal gray matter and corpus stratium. The abnormal cells analyzed included samples of anaplastic astrocytoma, pilocytic astrocytoma and oligodendroglioma. Figure 14. Average cell nucleus size vs. slide sample Figure 14 shows the statistics obtained concerning the average cell nucleus size. Again, a clear separation can be seen between the normal and abnormal cell samples with the exception of one of the anaplastic astrocytoma sample that is very close to the gray matter samples. Since cell segmentation was not optimized at this time, the error tends to be set toward the cells having a larger area than they really have due to improper cell segmentation. For comparison, the same slide images were analyzed with the optimized script. The results are given in the figures below. Figure 15. Optimized cell count vs. slide sample Figure 13. Graph of cell count vs. slide sample From Figure 15, it can be seen that the normal cell count is now between 200 and 400 cells per slide image and the abnormal ones range from 600 to 1400 cells. Here we have a 200-cell gap between the normal and abnormal ones as opposed to a 100-cell gap we had before. The optimized cell size graph also shows some interesting results.

7 Figure 16. Optimized cell nuclei size vs. slide sample Figure 16 shows that the average cell nucleus size for normal white matter ranges from 60 to 100 pixels, while the normal gray matter average is from 90 to 110 pixels. The gap between the average cell nucleus size of normal and abnormal cells is not that clear. The abnormal cell nucleus size are larger, but the average cell nucleus size is between 110 and 175 pixels. More samples need to be analyzed before more conclusive results can be obtained, but this can be used as a start to have a computer make a distinction between normal and questionable slide samples. Since 9 samples were used for white and gray matter, a clear pattern can be seen for the results. However, there were not enough samples of the other cell types to establish a pattern. Nevertheless, even with the small number of abnormal samples analyzed, it can clearly be seen that the cell density is considerably higher in abnormal cells than in normal cell samples. 6. Clinical Real-time Interactive Pathology The study presented in this paper shows some very interesting results in a field where very little research has been done. Computerized instruments have been developed and used in the areas of cytology and hematology using neural networks. In the case of hematology, instruments are available to count and recognize the number of red blood cells, white blood cells and platelets. This helps medical personnel look at large numbers of slides by letting a machine flag the slides that don t meet the normal specifications. This allows the trained professional to look at only those slides that exceed the normal thresholds, along with spot-checking the normal ones to make sure that the machine is correctly analyzing the normal samples. Hopefully, in the near future, this work can be expanded to develop a tool that can be used to analyze a slide sample real-time with the aid of a DSP board. This will allow statistical information to be provided and flagged as normal, or questionable, while a pathologist looks at it under a microscope. Some ideas for future enhancement of this work include: to give the user the capability to choose the grayscale threshold value to remove the background noise interactively, or by computer automatically with the help of histogram. This can be done by the addition of a slider bar with the range from 0 to 255 with a default setting of 200 (a static value that the script has at this time). Another enhancement could be to establish cutoff values between normal and abnormal values and have the results display in different colors depending on these values. For example, if the results have a cell count between 200 and 400, display the results in green, but if the results have a cell count above 600 cells, display the results in red, and then anything between 400 and 600, display them as yellow. At this time, the full image is analyzed and it takes less than one minute to analyze a picture of about 500KB. Another enhancement can be to select only a small region of interest to speed up the process. This would not be a problem if a DSP board, like TI TMS320DM642, can be integrated to perform the video capture and analysis, while MATLAB handles the user interface and communication with the DSP board [3]. The Image Acquisition Toolbox, that is available from Mathworks, can also be used to capture images directly from a digital camera connected to a microscope, thus eliminating the need to digitally photograph the slide sample and then send it to the c:\images directory of the computer for analysis. Since the pixel size is relative to the size of the image, there is no way to determine the actual cell nucleus size. We can only give a relative number to the size in pixels. This method works, as long as all the images are taken at the same resolution and they are not magnified differently. For this study, all images were taken at a 20x magnification. To solve this problem, a scale can be added to establish a physical size relationship in microns between actual nucleus size, magnification power, and pixel resolution, regardless if image is zoomed in or the common size relationship is lost between pictures. 7. References [1] Luis Hernandez, Image Analysis and Segmentation of Brain Cells, UHCL technical report, Fall 2003 [2] Mathworks MATLAB Reference Manuals with SDC Morphological Toolbox [3] TI Developers Conference Image Analysis & Segmentation presenters and presentation