Download Basic Principles of Image Processing

ANATOMIC PATHOLOGY
Review Article
Basic Principles of Image Processing
WENDY A. WELLS, M . R . C . P A T H . , ROBERT O. RAINER, M.D.,
AND VINCENT A. MEMOLI, M.D.
For many traditionally trained anatomic pathologists, the
concept of an image analysis system can be daunting. It
is difficult to overcome the initial skepticism that a machine can, in any way, duplicate the profound, natural
image analyzing processes of the human brain. When analyzing the amount of immunohistochemical staining in
a tissue section, a machine must be able to mimic the
many compensatory mechanisms of a trained professional
by simultaneously making allowances for process variables
such as section thickness variability, staining irregularities,
irrelevant background staining, and poor representation
of the lesion.
The advent of inexpensive microprocessors, high-quality cameras, and more affordable memory devices makes
image processing and statistical image analysis practical
and cost-effective. Advances in software development
make these technologies accessible and comprehensible
to operators with varied experience in applied computer
sciences.
The evaluation of any image analysis system requires
a basic background knowledge of image processing, including its distinct vocabulary, an understanding of its
advantages and limitations, its methods of standardization, and its applications. This information is more readily
available in the optical physics and engineering literature
than in the medical literature. In the latter, details of theory, equipment, and standardization, so helpful in understanding, comparing, and contrasting different image
analysis systems, is minimal. This deficiency will be addressed in two articles, the first concerning the basic principles of image processing, the second detailing necessary
equipment, standardization, and applications. With this
information, it is hoped that some of the technical myths
surrounding image analysis can be dispelled.
PRINCIPLES OF IMAGE PROCESSING
Image processing is the manipulation of pictorial information to enhance and evaluate maximally the visual
qualities of the original image. In this way, it is possible
to exaggerate certain details in the digitized image not
appreciated in the original form.
Until recently, the computer and statistical analysis involved in image processing was deemed too complicated
and time-consuming for routine use. The methods of
standardization were poorly defined and some of the detail
in the original color image was lost in the transformation
to a black-and-white (gray-value) digitized image.
Today, reasons for using a quantitative image analysis
system are that it is rapid, reliable, intuitive, and reproducible. To appreciate these qualities, certain basic principles must be emphasized.
First, to program a machine to reproduce the imageenhancing characteristics of the human visual system, the
latter must be defined and understood. The logarithmic
light response of the human eye, its Mach band, and simultaneous contrast effects will be discussed.
Second, an image analysis system, in its simplest form,
comprises a microscope, a video camera, a computer, and
a display screen (a cathode ray tube). To acquire the best
image possible for analysis on the display screen, the microscope must provide optimal, calibrated illumination.
Light scatter and light source irregularities must be eliminated. Camera sensitivity and resolution must be maximized. The conversion to a gray-value, digitized image
must be representative of the original color image. Techniques required to eliminate background noise, enhance
contrast,
and improve focus in the digitized image must
From the Department of Pathology, Dartmouth-Hitchcock Medical
Center. Lebanon. New Hampshire.
be understood and easily implemented. These features
will be discussed in further detail.
Supported in part by the Hitchcock Foundation when Dr. Wells was
a Tiffany Blake Fellow at the Dartmouth-Hitchcock Medical Center.
Received February 3, 1992; revised manuscript accepted for publication
March 3, 1992.
Address reprint requests to Dr. Memoli: Department of Pathology,
Dartmouth-Hitchcock Medical Center, Lebanon, New Hampshire 03756.
The Human Visual System
Knowledge of the natural image-enhancing characteristics of the human visual system may be helpful in un493
494
ANATOMIC PATHOLOGY
Review Article
LOGARITHMIC RESPONSE OF THE HUMAN EYE
White
V
*
Perceived
brightness
Black
White
Illumination
Intensity
FIG. 1. With a logarithmic response to perceived brightness, there is a
much greater change in the perceived brightness of darker regions than
lighter regions for the same change in illumination intensity. (Adapted
from Baxes GA. Digital Image Processing—The Basics. Englewood Cliffs,
NJ: Prentice-Hall, Inc., 1988.)
the relationship between illumination intensity on the rod
and cone photoreceptors and perceived brightness is logarithmic rather than linear. Thus for the same change in
illumination intensity, there is a much greater change in
the perceived brightness of darker regions in the image
than brighter regions. By simply darkening an image, previously undetected details can be illuminated (Fig. 1).
Second, the human eye displays a "simultaneous contrast effect" whereby the perceived brightness of an area
depends on the intensity of the surrounding area. Given
two identically sized images with the same gray-value intensity, the one with a black background will appear
brighter than the one with a white background (Fig. 2).
Third, the human visual system can accentuate sharp
intensity changes by employing the "Mach band effect."
At the immediate interface of a dark region and a light
region, the human eye perceives a more exaggerated
change in the brightness transition than what is actually
present (Fig. 3).
Direct comparisons can be made between these aspects
of image-contrast enhancement and those reproduced by
computer manipulation in an image-processing system.1
derstanding the computer manipulations required to reproduce similar qualities.
The human visual system routinely uses mechanisms
to enhance maximally details in its original image.' First,
MACH BAND EFFECT
SIMULTANEOUS CONTRAST EFFECT
Black
_,
Actual
brightness
GRAYVALUE
Perceived
brightness
White
FIG. 2. The perceived brightness of an area depends on the intensity of
the surrounding area; given two identically sized images with the same
gray value intensity, the one with the black background appears brighter
than the one with a white background. (Adapted from Baxes GA. Digital
Image Processing—The Basics. Englewood Cliffs, NJ: Prentice-Hall, Inc.,
1988.)
Position of Interface
FIG. 3. At the interface of the dark and light regions, the human eye
perceives a more exaggerated change in the brightness transition than
that which is actually present. (Adapted from Baxes GA. Digital Image
Processing—The Basics. Englewood Cliffs, NJ: Prentice-Hall, Inc., 1988.)
A.J.C.P. • November 1992
WELLS, RAINER, AND MEMOLI
Basic Principles o)
495
Processing
B1MODAL HISTOGRAM
Methods of Image Processing
Generally three manipulation techniques process and
operate on images: (1) optical manipulation as refined by
darkroom photography over many years; (2) electrical manipulation and analog processing, similar
to that seen in a television, in which the amplitude of
the voltage corresponds to the brightness of the processed image; and (3) computer manipulation with digital
processing.2
At a set sampling frequency, the analog video signal is
converted into a digital form comprising pixel units. Each
pixel is defined by its location and gray-value intensity.
The latter ranges from black to intermediate gray values
to white. Each pixel gray value may be manipulated by
an intensity transformation function (ITF) before being
converted back to a pulse of voltages to be displayed on
the computer monitor.
IMAGE
IMAGE HISTOGRAM
0
White
255
Bad,
Gray value
FlG. 4. The histogram pictorially represents the number of pixels that
appear in the image at each gray value. A tall, narrow histogram shape
and a broad, flattened histogram represent images of low and high contrast, respectively. (Adapted from Inove S. Video Microscopy, New York:
Plenum Press, 1986.)
Intensity Transformation Function or "Look-up
Table"
Once an image has been converted to a digital form, it
can be manipulated within the memory of the computer
in ways that affect the contrast and overall brightness of
the image. The ITF (also called the look-up table) specifies
the way in which the original image (Im 1) is transformed
into a new image (Im2). These computer manipulations
enhance the visual qualities of the image as it appears in
its new form by varying the gray values of individual pixels
but not the relative gray values of adjacent pixels.
The Histogram
A useful tool in image processing is the histogram that
pictorially represents the number of pixels that appear in
the image at each gray value.2 In Figure 4, the image comprises two areas, A and B, with different gray values, represented by a histogram as shown. A tall, narrow histogram would represent large numbers of pixels with equal
or nearly equal gray values. This reflects an image of low
contrast in which small points of detail are difficult to
differentiate. A broad histogram would represent pixels
of variable gray values. This reflects an image of high contrast in which the degree of image detail is markedly enhanced.
The ITF controls the shape of the histogram and hence
influences changes in image contrast. The ITF converts
the original image into a second, modified image. The
output gray values of the ITF are solely dependent on the
input gray values. Thus the relative information between
adjacent pixels remains the same. Therefore,
GV2 = f (GV1)
where GV1 = the input gray value of image 1 and GV2
= the output gray value of image 2. As a linear function,
GV2 = mGVl + b
where m = slope of the line (a steep slope indicates better
contrast) and
b = intercept of the gray-value axis.
A linear ITF with a fixed slope (m = 1) but variable
axis intercept (b value) is demonstrated in Figure 5. The
overall relationship of the individual pixels in the image
remains the same and so the shape of the input and output
histograms are identical. But when b >0, the gray value
of every pixel is decreased by the same amount and so
the transformed image appears universally paler. When
b <0, the gray value of every pixel is increased by the
same amount and so the transformed image appears universally darker.
Figure 6 demonstrates a linear ITF with a fixed axis
intercept (b = 0) but increased line slope (m >1). The
range of gray values displayed in the output histogram
image is then increased. This broadens the output histogram and increases the contrast of the output image. In
Figure 7, the axis intercept is fixed but the line slope is
decreased (m <1). The output histogram is narrowed,
representing a low-contrast output image.
The ITF does not have to be linear. A logarithmic function mimics the way most photographic processes and the
human visual system work.3 The slope of the nonlinear
Vol. 98 No. 5
ANATOMIC PATHOLOGY
496
Review Article
LINEAR INTENSITY TRANSFORMATION FUNCTION HTF)
(m=1)
OUTPUT HISTOGRAM
255
(Black)
•
/
)
) LIE
) m=1
) b=variable
Gray
values
FIG. 5. Linear intensity
transformation function with
a fixed-line slope (m = l)
but variable axis intercept
(b value). When b >0, the
transformed image appears
universally paler. When b
< 0, the transformed image
appears universally darker.
0
(White)
50
100
150
Gray values
200
200
255
(Black)
INPUT HISTOGRAM
function, in a semilog plot, is known as gamma. A positive
gamma will compress the histogram at the bright end while
expanding the dark end. Negative gamma values will have
the opposite effect on an image. A special transfer function
known as the histogram equilization method will remap
the pixels in such a way that an equal number of pixels
in the final image will have a given brightness value. The
shape of this function can be derived directly from the
shape of the original histogram.
Intensity (I) of staining is expressed in terms of average
gray levels but cannot be used to compare levels of staining
in different regions. For example, immunohistochemical
staining in a region with an average gray value of 100 is
not stained twice as heavily as a region with an average
gray value of 200.
Transmittance (T) is determined by the amount of regional staining, where T is the ratio of gray level (GL) in
the region of interest to that of the incident or blank field
light.
Intensity Versus Transmittance Versus Optical
Density
T = GLspecimen/ GLblank
A number of frequently used, and sometimes confusing,
terms must be defined and differentiated.4
Optical density (OD) is a logarithmic function of transmittance. It is used for two reasons. First, as discussed,
A.J.C.P. • November 1992
497
WELLS, RAINER, AND MEMOLI
Basic Principles of Image Processing
LINEAR INTENSITY TRANSFORMATION FUNCTION
cm
ill
OUTPUT HISTOGRAM
ITFH
255
(Black)
m>1
b=0
200
cm.
150
Gray values
FIG. 6. Linear intensity
transformation function with
a fixed axis intercept (b = 0)
but an increased line slope
(m >1). The output histogram is broadened, representing a high-contrast output image.
100
50
(White)
b=0
0
4.54 sq. cm.
No. of pixels
No. of
pixels
0
50
(White)
100
150
Gray values
200
255
(Black)
INPUT HISTOGRAM
the human eye also displays a logarithmic response to
changes in light brightness. Second, according to the BeerLambert law, the OD of a solution varies linearly with
concentration. Hence, a region of tissue displaying an OD
value of 1.0 will be twice as heavily stained as a region
with an OD of 0.5.
OD = -log (T)
OD = log (GLblank/GLspecimen)
pixels (bitmap) to output a different array of pixels. Most
image-processing algorithms are intended to modify the
processed image in such a way that a specific aspect of
the original image is enhanced, often at the expense of
others. Most often, the features filtered out from the background include edges, boundaries, a desired object, or
some other defined structure. The image-processing algorithms commonly used and described here are point
operations and segmentation.
Point Operations
IMAGE-PROCESSING ALGORITHMS
Many software algorithms have been developed to process an image with the aid of a microcomputer.3 Image
processing modifies the brightness value of an array of
These are the simplest techniques used in image processing. These software tools will replace the value of a
given pixel based solely on the previous value of that pixel.
Vol. 98 • No. 5
498
ANATOMIC PATHOLOGY
Review Article
LINEAR INTENSITY TRANSFORMATION FUNCTION
(Black)
255 H
OUTPUT HISTOGRAM
200 -
150
Gray
values
100
FIG. 7. Linear intensity
transformation function with
afixedaxis intercept (b = 0)
but a decreased line slope
(m < 1). The output histogram is narrowed, representing a low-contrast output
image.
50
(White)
0
(White)
100
150
200
Gray values
255
(Black)
INPUT HISTOGRAM
Their overall effect is only to alter the appearance of an
image and not to affect subsequent measurements. However, by enhancing the image projected on the computer
screen, previously hidden details are revealed. Examples
of point operations are the intensity transformation functions as previously discussed, smoothing (noise reduction),
and image sharpening.
Smoothing. These are algorithms used to correct defects
present in the image, commonly called "noise." Noise
may be either random (stochastic) or periodic. Most of
the random noise can be eliminated simply by averaging
many captured images. The problem pixels, which are
randomly present at different locations in each image, are
averaged and removed. It is important that the image is
stationary, as is the case with microscopic images, when
using averaging to reduce random noise.
If the random noise still persists in the image after averaging, a tool known as gaussian smoothing can be used.
This algorithm will replace the brightness value of each
pixel in the image with a value representing the average
of the central pixel and its surrounding eight neighbors.
The problem associated with this center-weighted
smoothing kernel method is that gaussian smoothing can
degrade the sharpness of edges and the image as a whole.
The best method to remove random noise is the median
filter.5 The brightness value of each pixel in the image
and its eight neighbors are ranked in order, and the median
value (the fifth brightest) is used to replace the original
central pixel. The disadvantage of the median is that it is
very time consuming, and this algorithm is often used as
a benchmark in assessing the speed performance of an
image-processing system.
A.J.C.P. • November I992
WELLS, RAINER, AND MEMOLI
499
Basic Principles Oj
Image Processing
The gaussian kernel and the median filter operate in
the spatial domain. They deal with each individual pixel
and its neighbors based on their physical or spatial relationship to each other. This class of operations is generally
ineffective in removing periodic noise. To delete this type
of noise, one must rely on processing operations that take
place in the frequency domain. The most familiar example
is the application of the Fourier transformation. The convolutions are carried out in the transformed image rather
than the original image. The latter can be recovered by
applying the inverse of the original transform function.
Frequency transformations are very demanding computations because they often rely on trigometric functions.
Sharpening. The Laplacian kernel is a nondirectional
second derivative that will not alter the pixel values in
uniform or smoothly varying regions, but will extenuate
regions of change, such as edges and lines. Areas of change
are highlighted, whereas the areas of uniformity are suppressed. This convolution kernel mimics the inhibition
used by the human visual system and responds strongly
to discontinuities in an image, regardless of their orientation. If a Laplacian image is added back to the original
image, the edges will be enhanced but the overall contrast
is markedly reduced. Thus the use of sharpening operators
should be used only to improve the visual appearances of
images and not as a precursor to improved processing.
Because the Laplacian filter is a high-pass filter, it is most
sensitive to points and least sensitive to edges. Thus it
may increase the amount of noise present in an image
due to isolated points.
Segmentation
Perhaps one of the most challenging aspects of image
processing is segmentation of the image into meaningful
data. For useful measurements to be obtained by imageprocessing techniques, the object of interest must be distinguished from the background. Because humans do this
very well, it is efficient for an operator to outline the object
with a pointing device. Object segmentation by a computer is performed using two general principles. In one,
the object of interest can be found by discovering areas
where pixel values are homogenous. In another, when
objects do not differ appreciably from their surroundings,
one must rely on edge detection.
Edge Detectors. An edge can be defined as an area that
corresponds to a sudden shift from one pixel value to
another. As discussed earlier with the Laplacian filter, one
can scan the image looking for changes in the brightness
derivatives to discern an edge.
The Robert's cross-edge operator is an early example
of an algorithm that delineates edges but does not change
the original image.6 Two brightness derivatives, obtained
at right angles to each other and each orientated at 45
degrees to the pixel grid, are used to determine the magnitude of the slope change. This method has the added
benefit of providing information regarding the direction
of the edge but may be sensitive to any noise that is
present.
The Sorbel and Kirsh operators also are used as edge
detectors.7 These algorithms use kernels similar to the
Laplacian filter, but they are more sensitive to edges than
points. The operator derivatives comprise 3 X 3 grids representing a total of nine pixels. For each operator grid,
the brightness value of the central pixel is the sum of the
surrounding eight pixel values. By rotating this operator
grid on pixels throughout the image, the number of edges
found in a given direction can be graphed.
The Robert's cross-edge and Sorbel and Kirsh operators
are applied to the image globally. An algorithm applied
locally in the detection of edges is known as "edge following."8 To segment the edge from the background, this
algorithm identifies the neighboring pixels in the area that
represents the path of an edge. This procedure is repeated
along the edge ridge border.
Thresholding. Unlike edge-detection algorithms, which
perform multiple operations on each pixel, methods that
segment an image based on pixel values proceed more
quickly because they work on the entire image at once.
Thresholding refers to the segmentation of a single or
known range of gray values within the image and will
discriminate objects of interest based on their brightness
relative to each other. This powerful tool is easily applied
in many instances, such as identifying numbers of mitoses
or assessing the distribution of immunohistochemical
staining. However, in many biologic specimens, the inherent contrast is low and the structures present exhibit
a similar tendency to absorb and scatter light. In these
cases, discussed in more detail later, varying counterstains
and complementary color filters are used to enhance the
image for analysis.
The threshold range that identifies the area of interest
in an image is selected by the observer by eye. Thus the
reproducibility among different observers may vary. Even
so, this may be the best method of thresholding at this
time and with the available technology.
Another method is to evaluate the pixel value histogram. In the ideal situation, the background pixels will
normally form one peak and the objects of interest will
form another smaller peak that can be used to set the
threshold values. However, very few images present with
this ideal histogram. Some automatic thresholding algorithms do exist, but they are not perfect.
Binary Images. Despite all of the previously mentioned
image-processing tools, with few exceptions, the image is
still not fully segmented for obtaining measurements. Be-
Vol.9 •No. 5
ANATOMIC PATHOLOGY
500
Review Article
cause many objects may still overlap, many measurements
are benefited if performed on a binary image. A binary
image converts all the pixels present in the threshold range
to the maximum pixel value ("on"), and all the pixels
outside this threshold range to the minimum pixel value
("off"). Binary image manipulation, unlike the previous
image-processing tools, works directly on the image itself.
The simplest technique used on binary images is to
combine them logically into a single image. Boolean type
operators3 are "AND," "OR," "EX-OR" (exclusive OR),
and "NOT." The NOT statement simply negates the previous operation. All the pixels previously turned on are
turned off. The AND statement combines two images,
emphasizing features that are shared by both images. The
OR statement is used when combining two images acquired by implementing different procedures, such as differing threshold ranges. The EX-OR operation of two images gives a result in which pixels are turned on when
they are ON in either original image but not both.
Another major class of operators are the neighbor operators.9 Many of these operators respond to the feature
in or the shape of the object in question. Erosion is an
example of a neighboring operator. Each binary pixel is
examined, and if any of its neighbors are "off," the pixel
itself is turned "off." The net effect is to reduce the features
around the periphery of an object. Complementary to
this process is dilatation, where the periphery of an object
will be added. These two simple operators are very useful
when used in combination, and they represent another
set of operators known as "opening" and "closing."
A process useful in separating touching features is
known as "skeletonization" or medial axis transformation.
Fractal dimensions of binary images are useful in determining the length of the perimeter of the object.
Applications of these algorithms include separating and
individually counting touching cells as well as analyzing
surface cytoplasmic immunoreactivity staining as distinct
from nuclear.
IMAGE ANALYSIS ALGORITHMS
Basic, but most useful, measurements used in the routine analysis of tissue sections include the following.4
Area
Measurements
Area measurements simply correlate the area of interest
to a recorded number of pixels represented in that area.
A calibration function in the software will relate pixel
number to calibrated units, such as square millimeters or
centimeters and so on. Many areas of interest in the digitized image, such as mitoses, individual nuclei, and immunohistochemical staining, can be identified by thresholding or segmenting a known range of gray values. Other
more irregular shapes can be evaluated with an outlining
drawing device used manually by the operator.
Area Fraction
The area fraction (Aa) describes the relationship between the total area of interest (At) and the number of
pixels thresholded within this area (Ap).
Aa = Ap/At
Gray-Level
Measurement
Gray-level distribution in a stained area enables a relative staining intensity to be compared in different regions
of the same tissue. This is particularly useful in the assessment of immunohistochemical staining.
The mean gray-value is calculated by dividing the sum
of all the gray level values in the thresholded area by the
number of pixels that compose this area.
However, unless sources of variation within the specimen are eliminated, such as tissue antigenicity, section
thickness, and background staining, the measurement of
mean gray value is meaningless. The standardization of
equipment and technical procedures will be discussed
later. Although every effort should be made to control
these parameters, this may not always be possible. In these
conditions, relative optical density measurements can be
made by comparing the optical density of specific and
background staining as well as that in a blank image without tissue. By subtracting the blank image, variations in
local light-source intensity can be identified. In an immunohistochemically stained slide, the specific staining
depends on the antigenicity of the tissue; the same variables, such as tissue fixation or section thickness, will exist
in both the areas with specific staining and those with
background staining. The computer subtracts the average
background gray level from the average gray level of the
stained region to give a measurement for the specific
staining.
Relative OD: Ratio of the OD of the area of interest to
the OD of the corresponding background (control)
= log(GLblank/GLspec.) /log(GLblank/GLback.)
Adjusted OD: Difference between the OD of the area
of interest and the OD of the background (control)
= ODspec. - ODback.
= log(GLblank / GLspec.) - log(GLblank / GLback)
CONCLUSIONS
With the advent of highly sophisticated and affordable
microprocessors, cameras, and microcomputers, image
analysis of microscopic images in the medical field provides a means to quantify, in a small way, the complex,
A.J.C.P. • N<ivember 1992
WELLS, RAINER, AND MEMOLI
•nage Processing
Basic Principles Oj
natural image-processing capabilities of the human brain.
Image processing is considered cost-effective, accurate,
labor-saving, and reproducible. Particularly in thefieldof
three-dimensional imaging, the new software techniques
for image processing, analysis, and feature discrimination
are continuously developing. This software can be used
by operators with only limited knowledge of the background theories involved. But to appreciate and usefully
implement the many applications of an image analyzer,
it helps to understand the distinct vocabulary, basic algorithmic tools, and limitations of image processing.
REFERENCES
1. Baxes GA. Digital Image Processing: A Practical Primer. Englewood
Cliffs, NJ: Prentice-Hall, Inc., 1988.
501
2. Inoue S. Video Microscopy, First Edition. New York: Plenum Press,
1986.
3. Russ JC. Computer-Assisted Microscopy: The Measurement and
Analysis of Images, Second Edition. New York: Plenum Press,
1990.
4. Conn PM. Quantitative and qualitative microscopy. In: Conn PM,
ed. Methods in Neurosciences. New York: Academic Press, 1990,
P3.
5. Russ JC. Image processing for the location and isolation of features.
In: Russ JC, ed. Microbeam Analysis. San Francisco: San Francisco Press, 1986, p 501.
6. Pratt WK. Digital Image Processing. New York: John Wiley, 1978.
7. Sobel I. Camera Models and Machine Perception, AIM-21. Palo Alto,
CA: Stanford Artificial Intelligence Lab, 1970.
8. Ballard DH, Brown CM. Computer Vision. Englewood Cliffs, NJ:
Prentice-Hall, 1982.
9. Levialdi S. Neighborhood operators: An outlook in pictorial data
analysis. In: Haralick RM, ed. Proceedings of the 1982 Nato Advanced Study Institute, Bonas, France. New York: Springer-Verlag, 1983, pp 1-4.