Download Multispectral Scleral Patterns for Ocular Biometric Recognition

This article appeared in a journal published by Elsevier. The attached
copy is furnished to the author for internal non-commercial research
and education use, including for instruction at the authors institution
and sharing with colleagues.
Other uses, including reproduction and distribution, or selling or
licensing copies, or posting to personal, institutional or third party
websites are prohibited.
In most cases authors are permitted to post their version of the
article (e.g. in Word or Tex form) to their personal website or
institutional repository. Authors requiring further information
regarding Elsevier’s archiving and manuscript policies are
encouraged to visit:
http://www.elsevier.com/copyright
Author's personal copy
Pattern Recognition Letters 33 (2012) 1860–1869
Contents lists available at SciVerse ScienceDirect
Pattern Recognition Letters
journal homepage: www.elsevier.com/locate/patrec
Multispectral scleral patterns for ocular biometric recognition
Simona Crihalmeanu ⇑, Arun Ross
West Virginia University, Morgantown, WV 26506, United States
a r t i c l e
i n f o
Article history:
Available online 19 November 2011
Keywords:
Biometrics
Sclera
Conjunctival vasculature
Multispectral
Ocular biometrics
a b s t r a c t
Biometrics is the science of recognizing people based on their physical or behavioral traits such as face,
fingerprints, iris, and voice. Among these characteristics, ocular biometrics has gained popularity due to
the significant progress made in iris recognition. However, iris recognition is unfavorably influenced by
the non-frontal gaze direction of the eye with respect to the acquisition device. In such scenarios, additional parts of the eye, such as the sclera (the white of the eye) may be of significance. In this article, we
investigate the use of the sclera texture and vasculature patterns evident in the sclera as a potential biometric. Iris patterns are better discerned in the near infrared spectrum (NIR) while vasculature patterns
are better discerned in the visible spectrum (RGB). Therefore, multispectral images of the eye, consisting
of both NIR and RGB channels, are used in this work in order to ensure that both the iris and the vasculature patterns are imaged. The contributions of this work include: (a) the assembling of a multispectral
eye database to initiate research on this topic; (b) the design of a novel algorithm for sclera segmentation
based on a normalized sclera index measure; and (c) the evaluation of three different feature extraction
and matching schemes on the assembled database in order to examine the potential of utilizing the sclera
and the accompanying vasculature pattern as biometric cues. Experimental results convey the potential
of this biometric in an ocular-based recognition system.
Ó 2011 Elsevier B.V. All rights reserved.
1. Introduction
Biometrics is the science of establishing human identity by using
physical or behavioral traits such as fingerprints, face, iris and voice
(Jain et al., 2004, 2008; Ross et al., 2006). The use of the ocular region
as a biometric cue gained prominence with the introduction of automated iris recognition systems in the mid ’90s (Daugman, 2004;
Ross, 2010; Bowyer et al., 2008) followed by the more recent exploitation of the periocular region (Park et al., 2011; Woodard et al.,
2010). The possibility of using the retinal blood vessels in the back
of the eye for human identification has also been explored (Hill,
1999) with varying degrees of success. This work attempts to establish the utility of another ocular trait – the sclera. The use of the
sclera as a biometric is especially significant in the context of iris
recognition, when changes in the gaze angle of the eye can result
in non-frontal iris images that cannot be easily recognized (Schuckers et al., 2007). By utilizing the texture of the sclera along with
the vascular patterns evident on it, the performance of an iris
recognition system can potentially be improved. However, in order
to establish this, it is necessary to first assess if sufficient
⇑ Corresponding author. Address: West Virginia University, Lane Dept. of Comp.
Sci. and Elec. Engineering, Rm. 173 ESB, P.O. Box 6109, Morgantown, WV 26506,
United States. Tel.: +1 304 293 2028; fax: +1 304 293 8602.
E-mail addresses: [email protected] (S. Crihalmeanu), Arun.
[email protected] (A. Ross).
URL: http://www.csee.wvu.edu/~simonac (S. Crihalmeanu).
0167-8655/$ - see front matter Ó 2011 Elsevier B.V. All rights reserved.
doi:10.1016/j.patrec.2011.11.006
discriminatory information can be gleaned from the sclera and the
accompanying vasculature pattern. This work designs an image processing and pattern recognition module for evaluating the potential
of the scleral biometric. The sclera (Trier, 2006; Kanski, 2003) is a
firm dense membrane composed of collagen and elastic fibers, with
thickness varying from 0.8 mm to 0.3 mm and organized in four layers: episclera, stroma, lamina fusca and endothelium. The sclera is
avascular, except in its outer surface, the episclera, which contains
the blood vessels nourishing the sclera. The anterior part of the
sclera, up to the edge of the cornea (the sclero-corneal junction)
and the inside of the eyelids, are covered by the conjunctival membrane, a thin layer containing secretory epithelium that helps lubricate the eye for eyelid closure, and protects the ocular surface from
bacterial and viral infections. The part of the conjunctiva that covers
the inner lining of the eyelids is called palpebral conjunctiva. The
outer surface of the eyeball is covered by the bulbar conjunctiva,
and the junction of the palpebral conjunctiva and the bulbar conjunctiva is called the conjunctival fornix. Bulbar conjunctiva is semitransparent and colorless, and contains blood vessels. Anatomically,
the blood vessels in bulbar conjunctiva can be differentiated from
those of episclera; while conjunctival vessels may slightly move because of the conjunctival membrane, those in episclera will not
(Heath, 2006). The rich vasculature patterns revealed in the episclera and conjunctival membrane are together referred to as conjunctival vasculature in this article. Acquiring and processing
images of the sclera surface and the conjunctival vasculature patterns is challenging due to the approximately spherical surface of
Author's personal copy
S. Crihalmeanu, A. Ross / Pattern Recognition Letters 33 (2012) 1860–1869
the eyeball, external lighting conditions, anatomical structure of the
layers of the sclera, and the fineness of vascular patterns. In order to
image the scleral surface with its blood vessel patterns, a sufficient
amount of light has to be directed into the eye. This light will be incident on the spherical surface of the sclera after passing through the
conjunctiva. Optical processes such as reflection and refraction are
difficult to control resulting in specular reflections with a variety
of different intensity values, topologies, shapes, sizes and locations.
The variations in the specular reflection are accentuated by the rapid
movement of the eyeball. Based on the angle of the incoming ray of
light, some specular highlights may have intensities closer to the
background intensity values; hence isolating the specular reflections can be difficult. Further, due to the low contrast between the
blood vessels in the foreground and the texture in the background,
classical methods of segmentation based on edge detection cannot
successfully segment the vasculature. Therefore, alternative methods that overcome the problem of intensity variations have to be
used. Upon rubbing the eye, the conjunctiva can slightly move relative to the sclera surface generating small deformations of its blood
vessels. Matching images of the conjunctival vasculature is greatly
impacted by these deformations. Medical conditions can also influence the performance of the segmentation and feature extraction
algorithms in all processing steps. Diabetes, for example, is associated with the loss of capillaries, macrovessel dilation and changes
in tortuosity in conjunctiva (Owen et al., 2005, 2008). Hyperemia,
conjunctivitis and hemorrhage influence the amount of blood vessels seen on the sclera surface, and affect the contrast between the
blood vessels and the background (the eye becomes reddish) (Heath,
2006). Age is another factor that influences the appearance of the
sclera surface. Conjunctiva can stretch and loosen from the underlying sclera leading to the formation of conjunctival folds. The white of
the eye becomes yellowish and an increase in melanin in conjunctiva can also appear. In younger people, blood vessels may be less
visible. With age, the conjunctival vasculature becomes thicker
and more visible.
The sclera texture is a relatively new area of research in biometrics (Derakhshani and Ross, 2007; Derakhshani et al., 2006; Crihalmeanu et al., 2009; Zhou et al., 2011). Therefore, it is essential to
consider different feature extraction and matching techniques in
order to determine an effective method to characterize this biometric. In this work, we consider some techniques that utilize
the entire sclera surface and others that consider only the conjunctival vasculature patterns for matching.
The article is organized as follows: Section 2 describes multispectral data acquisition, the process of image denoising and specular reflection detection and removal; Section 3 discusses sclera
segmentation; Section 4 describes the methods used for feature
extraction and matching, and Section 5 presents the experimental
results evaluating the recognition accuracy.
2. Processing scheme
While iris patterns are better observed in the near infrared (NIR)
spectrum, vasculature patterns are better observed in the visible
spectrum (RGB). Therefore, multispectral images of the eye can
potentially be used for combining the iris biometric with the scleral
surface and the conjunctival vasculature patterns for enhanced recognition. This article focuses only on the processing and matching of
multispectral sclera surface and the conjunctival vasculature. The
block diagram of the proposed system is shown in Fig. 1.
2.1. Image acquisition
Multispectral imaging captures the image of an object at multiple spectral bands often ranging from the visible spectrum to the
1861
infra-red spectrum. The small visible spectrum band (Gonzales
and Woods, 2001) is represented by three narrow sub-bands called
the red, green and blue channels, that range from 0.4 lm to 0.7 lm.
The infrared spectrum is divided into NIR (near infrared), MIR
(midwave infrared), FIR (far infrared) and thermal bands, ranging
from 0.7 lm to over 10 lm.
In this work, images of the eye were collected using the Redlake
(DuncanTech) MS3100 multispectral camera.1 The camera has
three array sensors based on CCD technology. Between the lenses
and the sensors there is a color-separating prism to split the ingoing
broadband light into three optical channels. The camera acquires
imagery of four spectral bands from a three channel optical system.
The CIR (color Infrared)/RGB configuration outputs three channels
represented as a 2D matrix of pixels that are stacked on top of each
other along the third dimension; the three channels correspond to
the near-infrared (NIR), red component, and a Bayer mosaic-like pattern (Bayer, 1976). Fig. 2 shows an example of a CIR image along
with its components. The first channel – the NIR component – is
stored as a separate image. The second channel – the red component
– is stored as the red component of the RGB image. The green and
blue components are obtained from the third channel of the CIR/
RGB configuration through a Bayer pattern demosaicing algorithm.2
The red pixels on the Bayer color array are ignored. As specified by
the sensor manufacturer, the center wavelength of each spectral
band is as follows: blue – 460 nm, green – 540 nm, red – 660 nm,
and NIR – 800 nm.
The interface used to collect multispectral images is composed
of an ophthalmologist’s slit-lamp mount and a light source. The
mount consists of a rest chin to position the head and a mobile
arm to which the multispectral camera is attached. While the person is gazing to the left or to the right, Fig. 3, the camera can be easily manipulated to focus on the white of the eye. The light source
illuminates the eye using a spectral range from 350 nm to
1700 nm, and is projected onto the eye via an optic fiber guide with
a ring light attached to its end. Due to the reflective qualities of the
eyeball, pointing a light source directly at the subject’s eye creates
a glare on the sclera. The issue is resolved by directing the light
source such that the incoming rays to the eyeball are approximately perpendicular to the pupil region. This is not always possible due to subtle movements of the eyeball. Thus, glare is not
always contained within the pupil region and may overlap with
the iris.
The multispectral camera generates images with a resolution of
1040 1392 3 pixels. The first 17 columns are removed due to
artifacts. The final size of the images is 1035 1373 3. Videos
of the right and left eye are captured from 103 subjects, with each
eye gazing to the right and to the left. Eight images per eye per
gaze direction are selected from the video, based on a proper illumination, less specular reflection and in focus images. The total
number of images is 3280. For one subject, only data from the right
eye was collected due to medical issues. Working with images
from the same video allows us to bypass some of the challenges
encountered by Crihalmeanu et al. (2009) primarily due to viewing
angle. The process of frame selection ensures that there is no
remarkable change in pose. Our multispectral collection contains
images of the eye with different iris colors.3 Based on the MartinSchultz scale,4 often used in physical anthropology, we classify the
images as light eyes (blue, green, gray), mixed eyes (blue, gray or
green with brown pigment, mainly around the pupil) and dark eyes
(brown, dark brown, almost black).
1
Hi-Tech Electronics, Spectral Configuration Guide for DuncanTech 3-CCD Cameras, http://www.hitech.com.sg.
2
RGB ‘‘Bayer’’ Color and MicroLenses, www.siliconimaging.com/RGBBayer.htm.
3
Data requests should be sent to [email protected].
4
http://wapedia.mobi/en/Eye_color.
Author's personal copy
1862
S. Crihalmeanu, A. Ross / Pattern Recognition Letters 33 (2012) 1860–1869
Fig. 1. Block diagram for enhancing and matching multispectral conjunctival vasculature.
Fig. 2. (a) Color-infrared image (NIR–Red-Bayer Pattern). (b) NIR component. (c) Red component. (d) Bayer pattern. (e) RGB image. (f) Green component. (g) Blue component.
(h) Composite image (NIR–Red–Green). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)
Fig. 3. (a) Right-eye-looking-left (R_L). (b) Right-eye-looking-right (R_R). (c) Left-eye-looking-left (L_L). (d) Left-eye-looking-right (L_R).
2.2. Image denoising
The denoising algorithm employed for the red, green, blue and
NIR components is the double-density complex discrete wavelet
transform (DDCDWT) presented in (Selenick, 2002). After denoising, all spectral components (NIR, red, green and blue) are geometrically resized by a factor of 1/3 resulting in a final size of
310 411.
Author's personal copy
S. Crihalmeanu, A. Ross / Pattern Recognition Letters 33 (2012) 1860–1869
2.3. Specular reflection
Specular reflections have to be detected and removed as they can
impact the sclera segmentation process (described in Section 3). As
stated earlier, the data acquisition system uses a light source to illuminate the eye region. The light directed to the eyeball generates
specular reflection that has (a) a ring-like shape, caused by the
shape of the source of illumination and (b) highlights, due to the
moisture of the eye and the curved shape of the eyeball. Both are detected and removed by a fast inpainting algorithm. In some images,
the ring-like shape may be an incomplete circle, ellipse, or an arbitrary curved shape with a wide range of intensity values. It may be
located partially in the iris region, making its detection and removal
more difficult especially since the iris texture has to be preserved as
much as possible. The specular reflections are detected using different intensity threshold values for each component: 0.60 for NIR,
0.50 for red and 0.80 for green. Only regions with less then 1000 pixels in size are labeled as specular reflection, and are morphologically
dilated and inpainted. In digital inpainting, the information from the
boundary of the region to be inpainted is propagated smoothly inside the region. The value to be inpainted at a pixel is calculated
using a PDE equation5 in which partial derivatives are replaced by finite differences between the pixel and its eight neighbors.
3. Sclera region segmentation
When the entire image of the eye is used for enhancing the conjunctival vasculature, it is difficult to distinguish between the different types of lines that appear in it: wrinkles, crows feet,
eyelashes, blood vessels. Therefore, a good segmentation of the
sclera region that clearly exhibits the blood vessels is necessary.
Even if the light is directed to the pupil region to avoid specular
reflections, the curved nature of the eyeball presents a wide variety
of intensity values across the sclera surface. Brighter skin regions
as a result of illumination, and occasionally the presence of mascara, will make the segmentation of the sclera along the contour
of the eyelid a challenging process. The algorithm to segment the
sclera region has three main steps as described below.
3.1. Coarse sclera segmentation: the sclera-eyelids contour
The method employed to segment the sclera region along the
eyelid contour is inspired by the work done in the processing of
LandSat imagery (Land + Satellite) (Tucker, 1979). A set of indices
are used to segment the vegetation regions in aerial multispectral
images. Similarly, the index that we use for coarse sclera segmentation is based on the fact that the skin has lesser water content
than the sclera, and hence exhibits a higher reflectance in NIR.
Since water absorbs NIR light, the corresponding regions based
on this index appear dark in the image. The algorithm is as follows:
1. Compute an index called the normalized sclera index
NSIðx; yÞ ¼ NIRðx;yÞGðx;yÞ
, where NIR(x, y) and G(x, y) are the pixel
NIRðx;yÞþGðx;yÞ
intensities of the NIR and green components, respectively, at
pixel location (x, y). The difference NIR G is larger for pixels pertaining to the sclera region; it is then normalized to help compensate for the uneven illumination. Fig. 4 displays the
normalized sclera index for all three categories as specified by
the Martin–Schultz scale: light colored iris, dark colored iris
and mixed colored iris.
2. Locate sclera by thresholding the NSI image with the threshold
value g = 0.1. Fig. 4(b) displays the scatter plot between the NIR
intensity values and the corresponding green intensity values
5
http://www.mathworks.com/matlabcentral/fileexchange/4551.
1863
for all pixels in the image. The pixels above the threshold
(g = 0.1) represent the background region while the rest represent the sclera region. Changing the value of g will modify the
slope of the boundary line between the pixels of the two segmented regions. The output of the thresholding operation is a
binary image.6 For each category in the Martin-Schultz classification, the largest connected region in the binary image is composed of sclera region only; or the sclera and the iris; or the
sclera and a portion of the iris. For dark irides (brown and dark
brown), the sclera region excluding the iris is localized
(Fig. 4(d) the first row, referred henceforth as IS). Thus, in this
case, further segmentation of the sclera and iris is not required.
For light irides (blue, green, etc.), regions pertaining to both the
sclera and iris are segmented (Fig. 4(d) the second row, referred
henceforth as IS). Here, further separation of the sclera and iris
is needed. For mixed irides (blue or green with brown around
pupil), the region of the sclera and the light colored portion of
the iris are segmented as one region (referred henceforth as IS).
The dark portion of the iris (brown) is not included (Fig. 4(d) the
third row). Here, further separation of the sclera and the portion
of the iris is needed. To finalize the segmentation of the sclera,
i.e., to find the boundary between the sclera and the iris regardless
of the color of the iris, the pupil is detected. The convex hull of the
segmented region IS and the pupil region will contain the sclera, the
pupil, and the iris or the portion of the iris. This region is referred to
as ISIP and is further processed. Since the proposed algorithm does
not deduce the color of the iris, it is applicable to all images irrespective of the eye color. As seen in Fig. 4(a), the location of the
pupil is also visible either as a dark region that does not overlap
the sclera region (in dark and mixed irides) or as a lighter disk
within the sclera region (in light irides). This information can be
exploited only if the color of the iris is known in advance. Therefore, in Section 3.2 we present an automatic way of finding the
pupil location regardless of the color of the iris.
3.2. Pupil region segmentation
The location of the pupil is needed to determine ISIP and to find
the boundary between the sclera and the iris regardless of the color
of the eye. The pupil segmentation method is described in (Crihalmeanu and Ross, 2011). In NIR images, the pupil region is characterized by very low intensity values and, by employing a simple
threshold, the pupil region is obtained. However, this isolates the
eyelashes as well. In order to isolate only the pupil, the difference
in intensity values between the pupil pixels and the rest of the pixels
in the image is increased by using the power-law transformation on
the NIR image: IPL ¼ c IxI , where c = 1 is a constant, IPL is the output
image, II is the input NIR image and x = 0.7. This image is further
binarized with the threshold 0.1. The resulting image IBW has the pupil and the eyelashes denoted by 1. On the other hand, the contour of
the sclera region from the convex hull of IS is detected. Using Hough
transform for line detection, the highest peak corresponding to the
longest line is selected and removed. An ellipse is fitted to the contour pixels and a mask defined as the pixels located within the ellipse is build. This ellipse mask is imposed on the binary image
IBW. The result is a binary image IP that will contain the pupil, and
possibly eyelashes as logical 1 pixels. Count the number N of
connected pixels in IP. Through an iterative process decrease the ellipse’s semi-major and semi-minor axis. With each decrease, construct new elliptical masks that when imposed on the binary
image IBW will render a smaller value for the number of connected
pixels N in IP. The connected object for N = 1 will correspond to
6
MathWorks, Image Processing Toolbox, Finding Vegetation in a Multispectral
Image, http://www.mathworks.com/products/image/demos.html.
Author's personal copy
1864
S. Crihalmeanu, A. Ross / Pattern Recognition Letters 33 (2012) 1860–1869
Fig. 4. Sclera region segmentation. The first row displays the results for dark colored iris, the second row displays the results for light colored iris, and the third row displays
the results for mixed colored iris: (a) The normalized sclera index (NSI). (b) NIR vs. green intensity values. (c) Histogram of the NSI. (d) Sclera mask contour imposed on
original composite image. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)
the location of the pupil. Fit a new ellipse to this connected object to
obtain the pupil mask. The determination of an accurate pupil
boundary is not necessary. Results are displayed in Fig. 5(a).
3.3. Fine sclera segmentation: the sclera-iris boundary
As mentioned in Section 3.1, the convex hull ISIP, of the segmented region IS and the pupil region (Fig. 5(b) and (c)) will contain the sclera, the pupil and the iris or the portion of the iris. A
finer segmentation of the iris is needed regardless of the color of
the eye. As in (Proenca, 2010), we define four measures called ‘‘proportion of sclera’’ p(x, y) in four directions: north, south, west and
east. In ISIP, the value of p(x, y) is set to 0 for all the pixels outside
the convex hull region. For a pixel (x, y) inside the convex hull,
the proportion of sclera in the north direction, p"(x, y), is computed
as the mean of all the pixels of column y above the location (x, y).
The proportion of sclera in the south direction, p;(x, y), is computed
as the mean of all the pixels of column y below the location (x, y).
The proportion of sclera in the west direction, p (x, y), is computed
as the mean of all the pixels along row x, left of the location (x, y)
and the proportion of sclera in the east direction, p?(x, y), is computed as the mean of all the pixels along row x, right of the location
(x, y). Fig. 6(a)–(c) illustrates this procedure.
We use the k-means clustering algorithm (k = 2) to segment the
iris, and find the limbus (sclera-pupil) boundary. The algorithm
uses the pixels contained within the segmented region ISIP as its input. Each pixel is viewed as a five-dimensional entity consisting of
the intensity value of the green component, intensity value of the
red component, the proportion of sclera in the north p"(x, y) and
south directions p;(x, y). According to the gaze direction – looking-to-the-left or looking-to-the-right – the proportion of the
sclera in the west or east direction as assessed in the red component is used as the fifth feature. To detect the direction of the gaze,
the y coordinate of the centroid of the segmented region IS and the
centroid of the pupil region is found and compared. For the left
gaze direction ypupil > ysclera and proportion of sclera in the east
direction is used; for the right gaze direction ypupil < ysclera and the
proportion of sclera in the west direction is used. Euclidean distances between the origin of the coordinate system and the centroid of each cluster are computed in order to determine the
label of the two clusters (the label can be ‘sclera’ or ‘iris’). The largest distance is associated with the sclera cluster; this is the white
region in Fig. 6(d). The smallest distance is associated with the iris
cluster; this is the black region in Fig. 6(d). Two binary images, a
mask for the sclera region and a mask for the iris region represent
the output. On examining the two binary masks, we observe that in
some images, the k-means algorithm erroneously labels portion of
the sclera as being the iris (mainly the corners of the sclera that are
less illuminated and have lower intensity values). To address this
issue, if the iris mask has more than one connected region, the region in the iris mask that overlaps with the pupil mask is assumed
to be the iris region, and is subtracted from the convex hull ISIP.
The algorithm failed to segment the sclera region properly for a
total of 151 images due to improper illumination or the presence of
plenty of mascara. A total of 3129 images corresponding to 99 subjects are further processed.
Author's personal copy
S. Crihalmeanu, A. Ross / Pattern Recognition Letters 33 (2012) 1860–1869
1865
Fig. 5. Sclera region segmentation. The first row displays the results for a dark colored iris, the second row displays the results for a light colored iris, and the third row
displays the results for a mixed colored iris: (a) Contour of the sclera (IS) and pupil. (b) Convex hull of the sclera and pupil mask ISIP. (c) Contour of the convex hull imposed on
the composite image.
4. Feature extraction and matching
4.1. Blood vessels enhancement
An examination of the three components of the RGB image suggests that the green component has the best contrast between the
blood vessels and the background. A selective enhancement filter
for lines, and implicitly for blood vessels, as described in (Qiang
et al., 2003) and similarly used in (Crihalmeanu et al., 2009) is applied to the green component (Fig. 7(a) and (b)).
We next design three different feature extraction and matching
methods. The first one is based on interest-point detection and
Fig. 6. Sclera region segmentation. The first row displays the results for dark colored iris, the second row displays the results for light colored iris, and the third row displays
the results for mixed colored iris: (a) Proportion of sclera in north direction p"(x, y). (b) Proportion of sclera in south direction p;(x, y). (c) The first and second row display the
proportion of sclera in east direction p (x, y) for left gaze direction and the third row displays the proportion of sclera in west direction p?(x, y) for right gaze direction. (d) Kmeans output. (e) Contour of the segmented sclera mask imposed on the composite image.
Author's personal copy
1866
S. Crihalmeanu, A. Ross / Pattern Recognition Letters 33 (2012) 1860–1869
Fig. 7. Blood vessel enhancement on the segmented sclera region: (a) green component of the segmented sclera. (b) Result of the enhancement of blood vessels. (c) Centerline
of the detected blood vessels. (d) Minutiae points: bifurcations (red) and endings (green). The output (the first 10 pairs of corresponding interest points) of the SURF algorithm
when applied to enhanced blood vessel images (the complement of the enhanced blood vessel images are displayed for better visualization): (e) Images of the same eye.
(f) Images of different eyes. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)
utilizes the entire sclera region including the vasculature pattern;
the second is based on minutiae points on the vasculature structure; and the third is based on direct correlation. While the first
two techniques do not need an explicit image registration scheme,
the third technique relies on image registration.
4.2. Speeded Up Robust Features (SURF)
The Speeded-Up Robust Features (SURF) algorithm (Bay et al.,
2008) is a scale and rotation invariant detector and a descriptor
of point correspondences between two images. These points called
‘‘interest points’’ are prominent structures such as corners and Tjunctions on the image. The algorithm uses a detector to locate
interest points that are represented using a feature descriptor.
The detector employs a Hessian matrix applied to the image convolved with Laplacian of Gaussian filters that further are approximated as box filters. These approximations allow the use of
integral images for image convolution as described in (Viola and
Jones, 2001). The scale space is divided into octaves and is analyzed
by up-scaling the filter size. The same image is convolved with a
filter of increasing size at a very low computational cost. Interest
points over multiple scales are localized using a non-maximum
suppression algorithm as described in (Neubeck and Gool, 2006).
The localized maximums are interpolated in scale and image space
using (Brown and Lowe, 2002). The descriptor uses the distribution
of intensity values in a square region of side equal to 20s, where s is
the scale centered at the interest point. This region is further split
into sub-regions. The entries for the 64 length feature vector are
the sum of the Haar wavelet responses on the horizontal and vertical directions in these sub-regions. In our work, SURF is applied to
the enhanced blood vessel images. The similarity between two
images is assessed using the Euclidean distance as a measure between their respective corresponding interest points. Only Euclidean distances greater than 0.1 are considered and the number of
corresponding interest point pairs is counted. Fig. 7 (e) displays
the first 10 corresponding interest points between images of the
same eye, and Fig. 7(f) between images of two different eyes.
4.3. Minutiae detection
Another technique to represent and match scleral images is
based on the cross-over points of the conjunctival vasculature. We
refer to these points as minutiae points based on the fingerprint biometric literature (Maltoni et al., 2003). Because of the large variations in intensity values and the low contrast between the blood
vessels and the background, classical methods of segmentation
based on edge detection are not robust and do not give good results.
We used region growing method for segmenting the enhanced
blood vessels based on the algorithm described in (Perez et al.,
1999). The labeling of each pixel as pertaining to the conjunctival
vasculature or background, is based on the information provided
by the intensity value and the magnitude of the gradient of the
pre-processed image. The result of conjunctival vasculature segmentation using region growing is a binary image, that is
subjected to morphological operations, mainly a thinning procedure through which the blood vessel thickness is reduced to one
pixel (Fig. 7(c)). The minutiae points, in this work, correspond to
the bifurcations of the centerline of the blood vessels (Fig. 7(d)).
Each blood vessel ramification has to be as least 4 pixels in length.
A point matching algorithm is used to compare the points
extracted from two images where each point is characterized as a
(x, y) location. The matching algorithm consists of finding an alignment between the two sets of points that will result in the maximum overlap of minutiae pairs from the two images. For two
minutiae sets, A = {a1, a2, a3, . . ., am}, ai = (xi, yi), i = 1. . .m and
B = {b1, b2, b3, . . ., bn}, bj = (xj, yj), j = 1. . .n, where m and n are the number of minutiae in A and B, minutiae ai is said to be in
correspondence with a minutiae bj if the Euclidean distance (E)
between them is smaller than a given tolerance t, i.e.,
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
Eðai ; bj Þ ¼ ððxj xi Þ2 þ ðyj yi Þ2 Þ 6 t. The match score is computed as the square of the number of corresponding minutiae points
divided by the product mn. The algorithm failed to detect minutiae
points on 28 images. This is due to the detection of blood vessels
without ramifications and blood vessels that do not intersect.
4.4. Direct correlation matching
In this method, the two images to be compared are first registered using an image alignment scheme, and a direct correlation
between corresponding pixels is then used to determine their similarity. Image registration is the process of finding a transformation
that aligns one image with another. The regions of the sclera in the
two images that are to be registered are cropped, and the images
Author's personal copy
1867
S. Crihalmeanu, A. Ross / Pattern Recognition Letters 33 (2012) 1860–1869
are padded to the same size. The image with the smaller height is
padded up and down with an equal number of rows. If the gaze
direction of the eye is to the left, the image with the smaller width
is padded to the right. If the gaze direction of the eye is to the right,
the image with the smaller width is padded to the left. To detect
the direction of the gaze, the y coordinate of the centroid of the
sclera region and the centroid of the pupil region is found and
compared. This process results in a better overlap of the two sclera
regions. The registration method used here, described in
(Periaswamy and Farid, 2003), models a local affine and a global
smooth transformation. It also accounts for contrast and brightness
variations between the two images that are to be registered. The
registration between two images, the source I(x, y, t) and the target
^; t 1Þ, is modeled by the transformation m
~ ¼ ðm1 ; m2 ; m3 ; m4 ;
Ið^
x; y
m5 ; m6 ; m7 ; m8 Þ: m7 Iðx; y; tÞ þ m8 ¼ Iðm1 x þ m2 y þ m5 ; m3 xþ m4 yþ
m6 ; t 1Þ, where m1, m2, m3, and m4 are the linear affine parameters, m5, m6 are the translation parameters, and m7, m8 are the contrast and brightness parameters. A multi-scale approach is
employed by using a Gaussian pyramid to downsample the images
~
to be registered. From a coarse-to-fine level, the transformation m
is determined globally at each level, and then locally, and the estimated parameters are used to warp the source image. Using the
linear affine parameters m1, m2, m3, and m4, and the translation
parameters m5, m6, the sclera mask of the source image is also registered. We use different measures to compare two registered
sclera images. These measures provide a quantitative score that
describes the degree of similarity or conversely the degree of error/distortion between two images. To generate genuine scores,
the measure is computed between pairs of images pertaining to
the same subject; to generate impostor scores, the measure is computed between the first image (from the set of eight sequence
images) of each subject pair. Having two registered images I1 and
I2, and the two sclera masks, mask1 and mask2, we assess the
T
similarity of the two images over the region mask1 mask2 using
different quantitative measures: root mean square error (RMSE)
(Wang and Bovik, 2009), cross-correlation (CORR), mutual
Table 1
The EER (%) results when using different correlation methods.
Performance measure
L_L
L_R
R_L
R_R
RMSE
CORR
MI
NMI
RIU
SSIM
1.51%
0.1%
2%
0.7%
4.6%
1.25%
4.75%
3.5%
5%
3.4%
6.25%
4.3%
5.6%
4.75%
6.4%
5.2%
7.5%
6%
2%
1.25%
2.57%
1.6%
5.1%
4%
Results are displayed using Receiver Operator Characteristic
(ROC) curves and normalized score histograms. The results indicate
lower EER values for left-eye-looking-left and right-eye-lookingright compared to left-eye-looking-right and right-eye-lookingleft. This is due to the curvature of the eyeball and to the fact that
facial features (such as the nose) partially obstruct the light directed to the left eye when looking right and the right eye when looking left.
5.1. SURF
The number of corresponding interest point pairs between
images of the same eye will generate a genuine score and the number of corresponding interest point pairs between images of different eyes will generate an impostor score. The ROC and the
normalized score distribution for both eyes, left and right gaze
direction were obtained. Fig. 8(a) displays the results for left-eyelooking-left. The approximate EER values are as follows: 0.37% for
left-eye-looking left, 1.7% for left-eye-looking-right, 1.25% for
right-eye-looking-left and 0.75% for right-eye-looking-right. Results indicate that SURF method distinguishes very well between
genuine and impostors.
5.2. Minutiae points
The ROC and the normalized score distribution for both eyes,
left and right gaze direction were obtained. As presented in
Fig. 8(b), an approximate EER value of 9.5% is obtained for lefteye-looking-left, 10.3% for left-eye-looking-right, 12% for righteye-looking-left, and 11.5% for right-eye-looking-right. A better
segmentation of the conjunctival vasculature, a better detection
of the finer blood vessels, and also a more accurate localization
of the centerline of the blood vessels, may improve the value of
EER when minutiae points are used for matching.
5.3. Direct correlation
The ROC and normalized score plots were obtained for all the
measures mentioned in Section 4.4, for both eyes, and both gaze
5
4
ROC Minutiae points
30
L_L EER~0.37%
L_R EER~1.7%
R_L EER~1.25%
R_R EER~0.75%
False Reject Rate (%)
False Reject Rate (%)
5. Results
ROC SURF
6
3
2
1
0
information (MI) (Sonka and Fitzpatrick, 2004), normalized mutual
information (NMI) (Maes et al., 1997), ratio-image uniformity (RIU)
(Sonka and Fitzpatrick, 2004), and structural similarity index
(SSIM) (Wang et al., 2004). The results are presented in Table 1.
L_L EER~9.5%
L_R EER~10.3%
R_L EER~12%
R_R EER~11.5%
25
20
15
10
5
0
1
2
3
4
False Accept Rate (%)
(a)
5
6
0
0
5
10
15
20
False Accept Rate (%)
25
30
(b)
Fig. 8. (a) The ROC curve for the SURF technique. (b) The ROC curve for the minutiae-based matching technique. (L_L: left-eye-looking-left; L_R: left-eye-looking-right; R_L:
right-eye-looking-left; R_R: right-eye-looking-right.)
Author's personal copy
1868
S. Crihalmeanu, A. Ross / Pattern Recognition Letters 33 (2012) 1860–1869
ROC Direct correlation, L_L
6
5
4
3
2
5
4
3
2
L_L EER~0.3%
L_R EER~3%
R_L EER~4.25%
R_R EER~1.25%
1
1
0
ROC Fusion CORR, Minutiae scores. Sum rule
6
CORR EER~0.1%
MI EER~2%
NMI EER~0.7%
RIU EER~4.6%
RMSE EER~1.51%
SSIM EER~1.25%
False Reject Rate(%)
False Reject Rate (%)
7
0
1
2
3
4
5
False Accept Rate (%)
6
7
(a)
0
0
1
2
3
4
False Accept Rate(%)
5
6
(b)
Fig. 9. (a) The ROC curve for the direct correlation method after processing the L_L images (left-eye-looking-left). (b) The ROC curve when the sum rule is used to fuse the
minutiae scores with CORR scores. (L_L: left-eye-looking-left; L_R: left-eye-looking-right; R_L: right-eye-looking-left; R_R: right-eye-looking-right.)
directions. The approximate values of EER are displayed in Table 1
(L_L – left-eye-looking-left, L_R – left-eye-looking-right, R_L –
right-eye-looking-left, R_R – right-eye-looking-right) and Fig. 9(a)
for L_L. The best performance is obtained when using the correlation measure, followed by the normalized mutual information
scheme. If the eye images with the exposed sclera region are taken
from approximately the same viewing angle, then direct correlation measures may be used as a matching method.
5.4. Score-level fusion
In our work, we used min–max technique to normalize and fuse
the minutiae scores with that of the direct correlation methods.
Correlation, mutual information, normalized mutual information,
and structural similarity index are similarity measures. Root mean
square error, and ratio intensity uniformity are dissimilarity measures. A dissimilarity score is transformed into a similarity score by
subtracting the normalized score from 1. Minutiae scores contain
information about the veins only, while the direct correlation
methods characterize the entire sclera surface. The results for score
level fusion using the sum rule, max rule and min rule (Ross et al.,
2006) shown in Table 2, indicate that for the fusion of minutiae
scores with CORR, RIU, SSIM, RMSE scores, sum rule and the max
rule perform the best. For the fusion of minutiae scores with MI
Table 2
The EER (%) results of the fusion of minutiae scores with different correlation
measures.
CORR
Sum rule
Max rule
Min rule
Sum rule
Max rule
Min rule
MI
L_R
L_L
L_R
L_L
L_R
0.3%
0.125%
9%
3%
3%
9.65%
3.5%
9%
2%
6.1%
9.2%
6.25%
4.4%
9.6%
0.525%
6.2%
10%
3.9%
R_L
R_R
R_L
R_R
R_L
R_R
4.25%
4.75%
11%
1.25%
1.25%
11%
7.2%
9.75%
7.5%
4.2%
7.95%
4.1%
7.95%
11.75%
5.9%
6%
11.5%
1.75%
RIU
Sum rule
Max rule
Min rule
Sum rule
Max rule
Min rule
NMI
L_L
RMSE
SSIM
L_L
L_R
L_L
L_R
L_L
L_R
3.1%
4.6%
9.6%
5.1%
6.25%
10%
1.5%
1.5%
9.25%
3.75%
4.75%
10.1%
1.65%
2.8%
3.5%
3.95%
4%
6.25%
R_L
R_R
R_L
R_R
R_L
R_R
7.25%
7.75%
11.9%
5.55%
5.1%
11.6%
5%
5.5%
12%
2.95%
2%
11.6%
5.9%
6.3%
7.9%
3.25%
4.5%
5.75%
scores, the sum and min rules have the best results, and for the fusion of minutiae scores with NMI scores, the min rule is the best
method. Fig. 9(b) shows the ROC curves after combining the minutiae and correlation scores using the sum rule for both eyes and
both gaze directions.
6. Discussion and conclusions
The article investigates the feasibility of using multispectral
conjunctival vasculature in an ocular biometric system. To remedy
the loss of information from off-angle and off-axis iris images,
additional details such as the sclera surface and its blood vessels
are exploited for identification. Iris patterns are better observed
in the NIR spectrum while the vasculature patterns are better
observed in the visible spectrum (RGB). Therefore, multispectral
images of the eye could be used for combining the iris biometric
with the sclera surface and its conjunctival vasculature patterns
for enhanced recognition. Being a relatively new research topic,
in this work we studied the challenges imposed when acquiring
and processing sclera images. A novel sclera segmentation method
was presented and different feature extraction and matching
techniques were used to represent this biometric. The matching
results using all the methods indicate lower values of EER for
left-eye-looking-left (L_L) and right-eye-looking-right (R_R) compared with right-eye-looking-left (R_L) and left-eye-looking-right
(L_R). This is because the facial features (such as the nose) partially
obstruct the light directed to the left eye when looking right and to
the right eye when looking left. In such cases the EER is improved
using fusion methods. The best accuracy (EER < 0.8% for L_L, R_R,
and EER < 1.8% for L_R, R_L) is obtained when interest-point detection (SURF) is used. This is because SURF utilizes the entire sclera
region including the vasculature patterns; further it is not sensitive
to small variations in the viewing angle, affine deformations, and
the color shades. Direct correlation measures also provide good results, mainly when correlation (EER < 1.3% for L_L, R_R, and
EER < 4.8% for L_R, R_L) and normalized mutual information
(EER < 1.7% for L_L, R_R, and EER < 5.3% for L_R, R_L) are used. Like
the SURF method, direct correlation measures use the entire sclera
surface including the conjunctival vasculature, but in contrast are
sensitive to changes in the viewing angle and illumination. Results
could be improved with a better blood vessel enhancement method. The minutiae based method presents an EER in the range
[9.5%,12%] for both eyes and both gaze directions. The performance
is greatly impacted by the presence of more or less prominent
veins, by the percentage of successfully segmented blood vessels
and the accuracy with which the centerline of the blood vessels
is found. The perceived shape and tortuosity of the blood vessels
Author's personal copy
S. Crihalmeanu, A. Ross / Pattern Recognition Letters 33 (2012) 1860–1869
are influenced by the small changes in the viewing angle. This
method can be improved by accurately segmenting the blood vessels resulting in a higher percentage of segmented blood vessels
containing finer veins. The fusion of direct correlation measure
scores with minutiae scores is performed in an attempt to boost
the matching performance. In the case of left-eye-looking-right
and right-eye-looking-left, where the light that reaches the eye is
obstructed by the facial structure and the curvature of the eyeball,
the EER is lowered when the sum rule is used for fusing minutiae
based scores with each of the direct correlation scores, except
when using mutual information and normalized mutual information. The results demonstrate the potential of using the sclera surface and the conjunctival vasculature for recognition and suggest
that more work is needed in this area. A future step is to address
the challenges encountered when acquiring and processing eye
images in order to design an improved ocular biometric system
by combining the iris with sclera surface patterns.
References
Bayer, B.E., 1976. Color imaging array. U.S.Patent 3 971 065.
Bay, H., Ess, A., Tuyleraars, T., Gool, L.V., 2008. Speeded-up robust features.
Computer Vision and Image Understanding (CVIU) 110 (3), 346–359.
Bowyer, K.W., Hollingsworth, K., Flynn, P.J., 2008. Image understanding for iris
biometrics: A survey. Computer Vision and Image Understanding 110 (2), 281–
307.
Brown, M., Lowe, D., 2002. Invariant features from interest point groups. In British
Machine Vision Conf., pp. 656–665.
Crihalmeanu, S., Ross, A., 2011. On the use of multispectral conjunctival vasculature
as a soft biometric. In: Proc. IEEE Workshop on Applications of Computer Vision
(WACV), Kona, USA, pp. 204–211.
Crihalmeanu, S., Ross, A., Derakhshani, R., 2009. Enhancement and registration
schemes for matching conjunctival vasculature. In: Proc. Internat. Conf. on
Biometrics, Alghero, Italy, pp. 1240–1249.
Daugman, J., 2004. How iris recognition works. IEEE Trans. Circuit Systems Video
Technol. 14 (1), 21–30.
Derakhshani, R., Ross, A., 2007. A texture-based neural network classifier for
biometric identification using ocular surface vasculature. In: Proc. Internat.
Joint Conf. on Neural Networks (IJCNN) Orlando, USA.
Derakhshani, R., Ross, A., Crihalmeanu, S., 2006. A new biometric modality based on
conjunctival vasculature. In: Proc. Artificial Neural Networks in Engineering,
St.Louis, MO.
Gonzales, R.C., Woods, R.E., 2001. Digital Image Processing, second ed. Tom
Robbins, Upper Saddle River, New Jersey.
Heath, G., 2006. The episclera, sclera and conjunctiva, Optometry today. Different.
Diagnosis Ocular Dis. 9 (2), 36–42.
Hill, R., 1999. Retina Identification. Biometrics: Personal Identification in Networked
Society. Kluwer Academic, pp. 123-141.
Jain, A., Ross, A., Prabhakar, S., 2004. An introduction to biometric recognition. IEEE
Trans. Circuits Systems Video Technol. 14 (1), 4–20.
Jain, A., Flynn, P., Ross, A., 2008, . Handbook of Biometrics, 1st ed., vol. 233. Springer
Science + Business Media, Spring Street, New York 10013, USA.
Kanski, J.J., 2003. Clinical Ophthalmology: A Systematic Approach. Elsevier Science
Limited.
1869
Maes, F., Collignon, A., Vandermeulen, D., Marchal, G., Suetens, P., 1997.
Multimodality image registration by maximization of mutual information.
IEEE Trans. Med. Imag. 16 (2), 187–198.
Maltoni, D., Maio, D., Jain, A., Prabhakar, S., 2003, . Handbook of Fingerprint
recognition, first ed., 233. Springer Science+Business Media, Inc.,, Spring Street,
New York, NY, USA.
Neubeck, A., Gool, L.V., 2006. Efficient non-maximum suppression. In: Proc. 18th
Internat. Conf. on Pattern Recognition (ICPR), vol. 3, pp. 850–855.
Owen, C.G., Newsom, R.S.B., Rudnicka, A.R., Ellis, T.J., Woodward, E.G., 2005.
Vascular response of the bulbar conjunctiva to diabetes and elevated blood
pressure. Ophthalmology 112 (10), 1801–1808.
Owen, C.G., Newsom, R.S.B., Rudnicka, A.R., Barman, S., Ellis, T.J., Woodward, E.G.,
2008. Diabetes and the tortuosity of vessels of the bulbar conjunctiva.
Ophthalmology 115 (6), e27–e32.
Park, U., Jillela, R., Ross, A., Jain, A.K., 2011. Periocular biometrics in the visible
spectrum. IEEE Trans. Inform. Forensics Security (TIFS) 6 (1), 96–106.
Perez, E.M., Hughes, A.D., Stanton, A.V., Tom, S.A., Bharath, A.A., Parker, K.H., 1999.
Segmentation of retinal blood vessels based on the second directional
derivative and region growing. In: Internat. Conf. on Image Processing (ICIP),
vol. 2, pp. 173–176.
Periaswamy, S., Farid, H., 2003. Elastic registration in the presence of intensity
variations. IEEE Trans. Med. Imag. 22 (7), 865–874, <http://
www.cs.dartmouth.edu/farid/research/registration.html>.
Proenca, H., 2010. Iris recognition: On the segmentation of degraded images
acquired in the visible wavelength. IEEE Trans. Pattern Anal. Machine Intell. 32
(8), 1502–1516.
Qiang, L., Shusuke, S., Kunio, D., 2003. Selective enhancement filters for nodules,
vessels, and airway walls in two or three dimentional CT scans. Med. Phys. 30
(8).
Ross, A., 2010. Iris recognition: The path forward. IEEE Comput., 30–35.
Ross, A., Nandakumar, K., Jain, A.K., 2006. Handbook of Multibiometrics. Springer
Publishers.
Schuckers, S.A.C., Schmid, N.A., Abhyankar, A., Dorairaj, V., Boyce, C.K., Hornak, L.A.,
2007. On techniques for angle compensation in nonideal iris recognition. IEEE
Trans. Systems Man Cybernet. B: Cybernet. 37 (5), 1176–1190.
Selenick, I.W., 2002. A new complex-directional wavelet transform and its
application to image denoising. In: IEEE Internat. Conf. on Image Processing,
vol. 3, pp. 573–576.
Sonka, M., Fitzpatrick, J.M., 2004. Image Registration. Handbook of Medical Imaging.
Medical Image Processing and Analysis, vol. 2. The International Society for
Optical Engineering (SPIE), pp. 487–497.
Trier, K., 2006. Advances in organ biology. The Sclera. The Biology of the Eye, vol. 10.
Elsevier Inc., pp. 353–373.
Tucker, C.J., 1979. Red and photographic infrared linear combinations for
monitoring vegetation. Remote Sens. Environ. 8, 127–150.
Viola, P.A., Jones, M.J., 2001. Rapid object detection using a boosted cascade of
simple features. Comput. Vision Pattern Recognition, 511–518.
Wang, Z., Bovik, A.C., 2009. Mean squared error: Love it or leave it? IEEE Signal
Process. Mag., 98–117.
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P., 2004. Image quality assessment:
From error measurement to structural similarity. IEEE Trans. Image Process. 13
(1), 600–612.
Woodard, D., Pundlik, S., Miller, P., Jillela, R., Ross, A., 2010. On the use of periocular
and iris biometrics in non-ideal imagery. In: Proc. Internat. Conf. on Pattern
Recognition (ICPR), Istanbul, Turkey, pp. 201–204.
Zhou, Z., Du, E.Y., Thomas, N.L., 2011. A comprehensive sclera image quality
measure. In: 11th International Conference on Control Automation Robotics
and Vision (ICARCV), pp. 638–643.