Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
ANATOMIC PATHOLOGY Review Article Basic Principles of Image Processing WENDY A. WELLS, M . R . C . P A T H . , ROBERT O. RAINER, M.D., AND VINCENT A. MEMOLI, M.D. For many traditionally trained anatomic pathologists, the concept of an image analysis system can be daunting. It is difficult to overcome the initial skepticism that a machine can, in any way, duplicate the profound, natural image analyzing processes of the human brain. When analyzing the amount of immunohistochemical staining in a tissue section, a machine must be able to mimic the many compensatory mechanisms of a trained professional by simultaneously making allowances for process variables such as section thickness variability, staining irregularities, irrelevant background staining, and poor representation of the lesion. The advent of inexpensive microprocessors, high-quality cameras, and more affordable memory devices makes image processing and statistical image analysis practical and cost-effective. Advances in software development make these technologies accessible and comprehensible to operators with varied experience in applied computer sciences. The evaluation of any image analysis system requires a basic background knowledge of image processing, including its distinct vocabulary, an understanding of its advantages and limitations, its methods of standardization, and its applications. This information is more readily available in the optical physics and engineering literature than in the medical literature. In the latter, details of theory, equipment, and standardization, so helpful in understanding, comparing, and contrasting different image analysis systems, is minimal. This deficiency will be addressed in two articles, the first concerning the basic principles of image processing, the second detailing necessary equipment, standardization, and applications. With this information, it is hoped that some of the technical myths surrounding image analysis can be dispelled. PRINCIPLES OF IMAGE PROCESSING Image processing is the manipulation of pictorial information to enhance and evaluate maximally the visual qualities of the original image. In this way, it is possible to exaggerate certain details in the digitized image not appreciated in the original form. Until recently, the computer and statistical analysis involved in image processing was deemed too complicated and time-consuming for routine use. The methods of standardization were poorly defined and some of the detail in the original color image was lost in the transformation to a black-and-white (gray-value) digitized image. Today, reasons for using a quantitative image analysis system are that it is rapid, reliable, intuitive, and reproducible. To appreciate these qualities, certain basic principles must be emphasized. First, to program a machine to reproduce the imageenhancing characteristics of the human visual system, the latter must be defined and understood. The logarithmic light response of the human eye, its Mach band, and simultaneous contrast effects will be discussed. Second, an image analysis system, in its simplest form, comprises a microscope, a video camera, a computer, and a display screen (a cathode ray tube). To acquire the best image possible for analysis on the display screen, the microscope must provide optimal, calibrated illumination. Light scatter and light source irregularities must be eliminated. Camera sensitivity and resolution must be maximized. The conversion to a gray-value, digitized image must be representative of the original color image. Techniques required to eliminate background noise, enhance contrast, and improve focus in the digitized image must From the Department of Pathology, Dartmouth-Hitchcock Medical Center. Lebanon. New Hampshire. be understood and easily implemented. These features will be discussed in further detail. Supported in part by the Hitchcock Foundation when Dr. Wells was a Tiffany Blake Fellow at the Dartmouth-Hitchcock Medical Center. Received February 3, 1992; revised manuscript accepted for publication March 3, 1992. Address reprint requests to Dr. Memoli: Department of Pathology, Dartmouth-Hitchcock Medical Center, Lebanon, New Hampshire 03756. The Human Visual System Knowledge of the natural image-enhancing characteristics of the human visual system may be helpful in un493 494 ANATOMIC PATHOLOGY Review Article LOGARITHMIC RESPONSE OF THE HUMAN EYE White V * Perceived brightness Black White Illumination Intensity FIG. 1. With a logarithmic response to perceived brightness, there is a much greater change in the perceived brightness of darker regions than lighter regions for the same change in illumination intensity. (Adapted from Baxes GA. Digital Image Processing—The Basics. Englewood Cliffs, NJ: Prentice-Hall, Inc., 1988.) the relationship between illumination intensity on the rod and cone photoreceptors and perceived brightness is logarithmic rather than linear. Thus for the same change in illumination intensity, there is a much greater change in the perceived brightness of darker regions in the image than brighter regions. By simply darkening an image, previously undetected details can be illuminated (Fig. 1). Second, the human eye displays a "simultaneous contrast effect" whereby the perceived brightness of an area depends on the intensity of the surrounding area. Given two identically sized images with the same gray-value intensity, the one with a black background will appear brighter than the one with a white background (Fig. 2). Third, the human visual system can accentuate sharp intensity changes by employing the "Mach band effect." At the immediate interface of a dark region and a light region, the human eye perceives a more exaggerated change in the brightness transition than what is actually present (Fig. 3). Direct comparisons can be made between these aspects of image-contrast enhancement and those reproduced by computer manipulation in an image-processing system.1 derstanding the computer manipulations required to reproduce similar qualities. The human visual system routinely uses mechanisms to enhance maximally details in its original image.' First, MACH BAND EFFECT SIMULTANEOUS CONTRAST EFFECT Black _, Actual brightness GRAYVALUE Perceived brightness White FIG. 2. The perceived brightness of an area depends on the intensity of the surrounding area; given two identically sized images with the same gray value intensity, the one with the black background appears brighter than the one with a white background. (Adapted from Baxes GA. Digital Image Processing—The Basics. Englewood Cliffs, NJ: Prentice-Hall, Inc., 1988.) Position of Interface FIG. 3. At the interface of the dark and light regions, the human eye perceives a more exaggerated change in the brightness transition than that which is actually present. (Adapted from Baxes GA. Digital Image Processing—The Basics. Englewood Cliffs, NJ: Prentice-Hall, Inc., 1988.) A.J.C.P. • November 1992 WELLS, RAINER, AND MEMOLI Basic Principles o) 495 Processing B1MODAL HISTOGRAM Methods of Image Processing Generally three manipulation techniques process and operate on images: (1) optical manipulation as refined by darkroom photography over many years; (2) electrical manipulation and analog processing, similar to that seen in a television, in which the amplitude of the voltage corresponds to the brightness of the processed image; and (3) computer manipulation with digital processing.2 At a set sampling frequency, the analog video signal is converted into a digital form comprising pixel units. Each pixel is defined by its location and gray-value intensity. The latter ranges from black to intermediate gray values to white. Each pixel gray value may be manipulated by an intensity transformation function (ITF) before being converted back to a pulse of voltages to be displayed on the computer monitor. IMAGE IMAGE HISTOGRAM 0 White 255 Bad, Gray value FlG. 4. The histogram pictorially represents the number of pixels that appear in the image at each gray value. A tall, narrow histogram shape and a broad, flattened histogram represent images of low and high contrast, respectively. (Adapted from Inove S. Video Microscopy, New York: Plenum Press, 1986.) Intensity Transformation Function or "Look-up Table" Once an image has been converted to a digital form, it can be manipulated within the memory of the computer in ways that affect the contrast and overall brightness of the image. The ITF (also called the look-up table) specifies the way in which the original image (Im 1) is transformed into a new image (Im2). These computer manipulations enhance the visual qualities of the image as it appears in its new form by varying the gray values of individual pixels but not the relative gray values of adjacent pixels. The Histogram A useful tool in image processing is the histogram that pictorially represents the number of pixels that appear in the image at each gray value.2 In Figure 4, the image comprises two areas, A and B, with different gray values, represented by a histogram as shown. A tall, narrow histogram would represent large numbers of pixels with equal or nearly equal gray values. This reflects an image of low contrast in which small points of detail are difficult to differentiate. A broad histogram would represent pixels of variable gray values. This reflects an image of high contrast in which the degree of image detail is markedly enhanced. The ITF controls the shape of the histogram and hence influences changes in image contrast. The ITF converts the original image into a second, modified image. The output gray values of the ITF are solely dependent on the input gray values. Thus the relative information between adjacent pixels remains the same. Therefore, GV2 = f (GV1) where GV1 = the input gray value of image 1 and GV2 = the output gray value of image 2. As a linear function, GV2 = mGVl + b where m = slope of the line (a steep slope indicates better contrast) and b = intercept of the gray-value axis. A linear ITF with a fixed slope (m = 1) but variable axis intercept (b value) is demonstrated in Figure 5. The overall relationship of the individual pixels in the image remains the same and so the shape of the input and output histograms are identical. But when b >0, the gray value of every pixel is decreased by the same amount and so the transformed image appears universally paler. When b <0, the gray value of every pixel is increased by the same amount and so the transformed image appears universally darker. Figure 6 demonstrates a linear ITF with a fixed axis intercept (b = 0) but increased line slope (m >1). The range of gray values displayed in the output histogram image is then increased. This broadens the output histogram and increases the contrast of the output image. In Figure 7, the axis intercept is fixed but the line slope is decreased (m <1). The output histogram is narrowed, representing a low-contrast output image. The ITF does not have to be linear. A logarithmic function mimics the way most photographic processes and the human visual system work.3 The slope of the nonlinear Vol. 98 No. 5 ANATOMIC PATHOLOGY 496 Review Article LINEAR INTENSITY TRANSFORMATION FUNCTION HTF) (m=1) OUTPUT HISTOGRAM 255 (Black) • / ) ) LIE ) m=1 ) b=variable Gray values FIG. 5. Linear intensity transformation function with a fixed-line slope (m = l) but variable axis intercept (b value). When b >0, the transformed image appears universally paler. When b < 0, the transformed image appears universally darker. 0 (White) 50 100 150 Gray values 200 200 255 (Black) INPUT HISTOGRAM function, in a semilog plot, is known as gamma. A positive gamma will compress the histogram at the bright end while expanding the dark end. Negative gamma values will have the opposite effect on an image. A special transfer function known as the histogram equilization method will remap the pixels in such a way that an equal number of pixels in the final image will have a given brightness value. The shape of this function can be derived directly from the shape of the original histogram. Intensity (I) of staining is expressed in terms of average gray levels but cannot be used to compare levels of staining in different regions. For example, immunohistochemical staining in a region with an average gray value of 100 is not stained twice as heavily as a region with an average gray value of 200. Transmittance (T) is determined by the amount of regional staining, where T is the ratio of gray level (GL) in the region of interest to that of the incident or blank field light. Intensity Versus Transmittance Versus Optical Density T = GLspecimen/ GLblank A number of frequently used, and sometimes confusing, terms must be defined and differentiated.4 Optical density (OD) is a logarithmic function of transmittance. It is used for two reasons. First, as discussed, A.J.C.P. • November 1992 497 WELLS, RAINER, AND MEMOLI Basic Principles of Image Processing LINEAR INTENSITY TRANSFORMATION FUNCTION cm ill OUTPUT HISTOGRAM ITFH 255 (Black) m>1 b=0 200 cm. 150 Gray values FIG. 6. Linear intensity transformation function with a fixed axis intercept (b = 0) but an increased line slope (m >1). The output histogram is broadened, representing a high-contrast output image. 100 50 (White) b=0 0 4.54 sq. cm. No. of pixels No. of pixels 0 50 (White) 100 150 Gray values 200 255 (Black) INPUT HISTOGRAM the human eye also displays a logarithmic response to changes in light brightness. Second, according to the BeerLambert law, the OD of a solution varies linearly with concentration. Hence, a region of tissue displaying an OD value of 1.0 will be twice as heavily stained as a region with an OD of 0.5. OD = -log (T) OD = log (GLblank/GLspecimen) pixels (bitmap) to output a different array of pixels. Most image-processing algorithms are intended to modify the processed image in such a way that a specific aspect of the original image is enhanced, often at the expense of others. Most often, the features filtered out from the background include edges, boundaries, a desired object, or some other defined structure. The image-processing algorithms commonly used and described here are point operations and segmentation. Point Operations IMAGE-PROCESSING ALGORITHMS Many software algorithms have been developed to process an image with the aid of a microcomputer.3 Image processing modifies the brightness value of an array of These are the simplest techniques used in image processing. These software tools will replace the value of a given pixel based solely on the previous value of that pixel. Vol. 98 • No. 5 498 ANATOMIC PATHOLOGY Review Article LINEAR INTENSITY TRANSFORMATION FUNCTION (Black) 255 H OUTPUT HISTOGRAM 200 - 150 Gray values 100 FIG. 7. Linear intensity transformation function with afixedaxis intercept (b = 0) but a decreased line slope (m < 1). The output histogram is narrowed, representing a low-contrast output image. 50 (White) 0 (White) 100 150 200 Gray values 255 (Black) INPUT HISTOGRAM Their overall effect is only to alter the appearance of an image and not to affect subsequent measurements. However, by enhancing the image projected on the computer screen, previously hidden details are revealed. Examples of point operations are the intensity transformation functions as previously discussed, smoothing (noise reduction), and image sharpening. Smoothing. These are algorithms used to correct defects present in the image, commonly called "noise." Noise may be either random (stochastic) or periodic. Most of the random noise can be eliminated simply by averaging many captured images. The problem pixels, which are randomly present at different locations in each image, are averaged and removed. It is important that the image is stationary, as is the case with microscopic images, when using averaging to reduce random noise. If the random noise still persists in the image after averaging, a tool known as gaussian smoothing can be used. This algorithm will replace the brightness value of each pixel in the image with a value representing the average of the central pixel and its surrounding eight neighbors. The problem associated with this center-weighted smoothing kernel method is that gaussian smoothing can degrade the sharpness of edges and the image as a whole. The best method to remove random noise is the median filter.5 The brightness value of each pixel in the image and its eight neighbors are ranked in order, and the median value (the fifth brightest) is used to replace the original central pixel. The disadvantage of the median is that it is very time consuming, and this algorithm is often used as a benchmark in assessing the speed performance of an image-processing system. A.J.C.P. • November I992 WELLS, RAINER, AND MEMOLI 499 Basic Principles Oj Image Processing The gaussian kernel and the median filter operate in the spatial domain. They deal with each individual pixel and its neighbors based on their physical or spatial relationship to each other. This class of operations is generally ineffective in removing periodic noise. To delete this type of noise, one must rely on processing operations that take place in the frequency domain. The most familiar example is the application of the Fourier transformation. The convolutions are carried out in the transformed image rather than the original image. The latter can be recovered by applying the inverse of the original transform function. Frequency transformations are very demanding computations because they often rely on trigometric functions. Sharpening. The Laplacian kernel is a nondirectional second derivative that will not alter the pixel values in uniform or smoothly varying regions, but will extenuate regions of change, such as edges and lines. Areas of change are highlighted, whereas the areas of uniformity are suppressed. This convolution kernel mimics the inhibition used by the human visual system and responds strongly to discontinuities in an image, regardless of their orientation. If a Laplacian image is added back to the original image, the edges will be enhanced but the overall contrast is markedly reduced. Thus the use of sharpening operators should be used only to improve the visual appearances of images and not as a precursor to improved processing. Because the Laplacian filter is a high-pass filter, it is most sensitive to points and least sensitive to edges. Thus it may increase the amount of noise present in an image due to isolated points. Segmentation Perhaps one of the most challenging aspects of image processing is segmentation of the image into meaningful data. For useful measurements to be obtained by imageprocessing techniques, the object of interest must be distinguished from the background. Because humans do this very well, it is efficient for an operator to outline the object with a pointing device. Object segmentation by a computer is performed using two general principles. In one, the object of interest can be found by discovering areas where pixel values are homogenous. In another, when objects do not differ appreciably from their surroundings, one must rely on edge detection. Edge Detectors. An edge can be defined as an area that corresponds to a sudden shift from one pixel value to another. As discussed earlier with the Laplacian filter, one can scan the image looking for changes in the brightness derivatives to discern an edge. The Robert's cross-edge operator is an early example of an algorithm that delineates edges but does not change the original image.6 Two brightness derivatives, obtained at right angles to each other and each orientated at 45 degrees to the pixel grid, are used to determine the magnitude of the slope change. This method has the added benefit of providing information regarding the direction of the edge but may be sensitive to any noise that is present. The Sorbel and Kirsh operators also are used as edge detectors.7 These algorithms use kernels similar to the Laplacian filter, but they are more sensitive to edges than points. The operator derivatives comprise 3 X 3 grids representing a total of nine pixels. For each operator grid, the brightness value of the central pixel is the sum of the surrounding eight pixel values. By rotating this operator grid on pixels throughout the image, the number of edges found in a given direction can be graphed. The Robert's cross-edge and Sorbel and Kirsh operators are applied to the image globally. An algorithm applied locally in the detection of edges is known as "edge following."8 To segment the edge from the background, this algorithm identifies the neighboring pixels in the area that represents the path of an edge. This procedure is repeated along the edge ridge border. Thresholding. Unlike edge-detection algorithms, which perform multiple operations on each pixel, methods that segment an image based on pixel values proceed more quickly because they work on the entire image at once. Thresholding refers to the segmentation of a single or known range of gray values within the image and will discriminate objects of interest based on their brightness relative to each other. This powerful tool is easily applied in many instances, such as identifying numbers of mitoses or assessing the distribution of immunohistochemical staining. However, in many biologic specimens, the inherent contrast is low and the structures present exhibit a similar tendency to absorb and scatter light. In these cases, discussed in more detail later, varying counterstains and complementary color filters are used to enhance the image for analysis. The threshold range that identifies the area of interest in an image is selected by the observer by eye. Thus the reproducibility among different observers may vary. Even so, this may be the best method of thresholding at this time and with the available technology. Another method is to evaluate the pixel value histogram. In the ideal situation, the background pixels will normally form one peak and the objects of interest will form another smaller peak that can be used to set the threshold values. However, very few images present with this ideal histogram. Some automatic thresholding algorithms do exist, but they are not perfect. Binary Images. Despite all of the previously mentioned image-processing tools, with few exceptions, the image is still not fully segmented for obtaining measurements. Be- Vol.9 •No. 5 ANATOMIC PATHOLOGY 500 Review Article cause many objects may still overlap, many measurements are benefited if performed on a binary image. A binary image converts all the pixels present in the threshold range to the maximum pixel value ("on"), and all the pixels outside this threshold range to the minimum pixel value ("off"). Binary image manipulation, unlike the previous image-processing tools, works directly on the image itself. The simplest technique used on binary images is to combine them logically into a single image. Boolean type operators3 are "AND," "OR," "EX-OR" (exclusive OR), and "NOT." The NOT statement simply negates the previous operation. All the pixels previously turned on are turned off. The AND statement combines two images, emphasizing features that are shared by both images. The OR statement is used when combining two images acquired by implementing different procedures, such as differing threshold ranges. The EX-OR operation of two images gives a result in which pixels are turned on when they are ON in either original image but not both. Another major class of operators are the neighbor operators.9 Many of these operators respond to the feature in or the shape of the object in question. Erosion is an example of a neighboring operator. Each binary pixel is examined, and if any of its neighbors are "off," the pixel itself is turned "off." The net effect is to reduce the features around the periphery of an object. Complementary to this process is dilatation, where the periphery of an object will be added. These two simple operators are very useful when used in combination, and they represent another set of operators known as "opening" and "closing." A process useful in separating touching features is known as "skeletonization" or medial axis transformation. Fractal dimensions of binary images are useful in determining the length of the perimeter of the object. Applications of these algorithms include separating and individually counting touching cells as well as analyzing surface cytoplasmic immunoreactivity staining as distinct from nuclear. IMAGE ANALYSIS ALGORITHMS Basic, but most useful, measurements used in the routine analysis of tissue sections include the following.4 Area Measurements Area measurements simply correlate the area of interest to a recorded number of pixels represented in that area. A calibration function in the software will relate pixel number to calibrated units, such as square millimeters or centimeters and so on. Many areas of interest in the digitized image, such as mitoses, individual nuclei, and immunohistochemical staining, can be identified by thresholding or segmenting a known range of gray values. Other more irregular shapes can be evaluated with an outlining drawing device used manually by the operator. Area Fraction The area fraction (Aa) describes the relationship between the total area of interest (At) and the number of pixels thresholded within this area (Ap). Aa = Ap/At Gray-Level Measurement Gray-level distribution in a stained area enables a relative staining intensity to be compared in different regions of the same tissue. This is particularly useful in the assessment of immunohistochemical staining. The mean gray-value is calculated by dividing the sum of all the gray level values in the thresholded area by the number of pixels that compose this area. However, unless sources of variation within the specimen are eliminated, such as tissue antigenicity, section thickness, and background staining, the measurement of mean gray value is meaningless. The standardization of equipment and technical procedures will be discussed later. Although every effort should be made to control these parameters, this may not always be possible. In these conditions, relative optical density measurements can be made by comparing the optical density of specific and background staining as well as that in a blank image without tissue. By subtracting the blank image, variations in local light-source intensity can be identified. In an immunohistochemically stained slide, the specific staining depends on the antigenicity of the tissue; the same variables, such as tissue fixation or section thickness, will exist in both the areas with specific staining and those with background staining. The computer subtracts the average background gray level from the average gray level of the stained region to give a measurement for the specific staining. Relative OD: Ratio of the OD of the area of interest to the OD of the corresponding background (control) = log(GLblank/GLspec.) /log(GLblank/GLback.) Adjusted OD: Difference between the OD of the area of interest and the OD of the background (control) = ODspec. - ODback. = log(GLblank / GLspec.) - log(GLblank / GLback) CONCLUSIONS With the advent of highly sophisticated and affordable microprocessors, cameras, and microcomputers, image analysis of microscopic images in the medical field provides a means to quantify, in a small way, the complex, A.J.C.P. • N<ivember 1992 WELLS, RAINER, AND MEMOLI •nage Processing Basic Principles Oj natural image-processing capabilities of the human brain. Image processing is considered cost-effective, accurate, labor-saving, and reproducible. Particularly in thefieldof three-dimensional imaging, the new software techniques for image processing, analysis, and feature discrimination are continuously developing. This software can be used by operators with only limited knowledge of the background theories involved. But to appreciate and usefully implement the many applications of an image analyzer, it helps to understand the distinct vocabulary, basic algorithmic tools, and limitations of image processing. REFERENCES 1. Baxes GA. Digital Image Processing: A Practical Primer. Englewood Cliffs, NJ: Prentice-Hall, Inc., 1988. 501 2. Inoue S. Video Microscopy, First Edition. New York: Plenum Press, 1986. 3. Russ JC. Computer-Assisted Microscopy: The Measurement and Analysis of Images, Second Edition. New York: Plenum Press, 1990. 4. Conn PM. Quantitative and qualitative microscopy. In: Conn PM, ed. Methods in Neurosciences. New York: Academic Press, 1990, P3. 5. Russ JC. Image processing for the location and isolation of features. In: Russ JC, ed. Microbeam Analysis. San Francisco: San Francisco Press, 1986, p 501. 6. Pratt WK. Digital Image Processing. New York: John Wiley, 1978. 7. Sobel I. Camera Models and Machine Perception, AIM-21. Palo Alto, CA: Stanford Artificial Intelligence Lab, 1970. 8. Ballard DH, Brown CM. Computer Vision. Englewood Cliffs, NJ: Prentice-Hall, 1982. 9. Levialdi S. Neighborhood operators: An outlook in pictorial data analysis. In: Haralick RM, ed. Proceedings of the 1982 Nato Advanced Study Institute, Bonas, France. New York: Springer-Verlag, 1983, pp 1-4.