Download B. Background and Significance

Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
A. Specific Aims
The primary goal of this project is to use biomedical engineering principles to develop generalpurpose software methods that can be integrated into complete therapy delivery systems. Such systems
will support more effective delivery of many image-guided procedures --biopsy, minimally invasive
surgery, and radiation therapy, among others. To understand the extensive role of imaging in the
therapeutic process, and to appreciate the current usage of images before, during, and after treatment, we
will focus our analysis on four main components of image-guided therapy (IGT): localization, targeting,
monitoring and control. We will use a succession of increasingly challenging testbeds to drive the
development of new science and technology.
The main focus of this partnership is the development and adaptation of robust algorithms. Clincal
project will be used to provide challenges, data sets and, ultimatley, validation of the overall utility of
the concpets developed.
Specifically, we will:
1. Develop robust algorithms for
 segmentation – automated methods that create patient-specific models of relevant anatomy
from multi-modal imagery.
 registration – automated methods that align multiple data sets with each other and with the
patient
2. Integrate these technologies into complete and coherent image guided therapy delivery systems;
3. Validate these integrated systems using performance measures established in particular application
areas.
To achieve these aims, we will create novel methods that build on our extensive ongoing research in
different areas of image guided procedures such as neurosurgery and prostate brachytherapy. Our goal is
to develop basic technology and to extend IGT capabilities into other applications, such as abdominal,
orthopedic and pelvic interventions. We anticipate encountering significant challenges in transferring
our experience with the brain into other body parts. As we develop solutions for each new anatomical
area, we expect that the performance specifications will change, requiring new techniques. For example,
we expect that we will have to extend rigid body registration to solutions that deal with deformable soft
tissue organs, for which more elaborate algorithms are needed. Hence, we will develop more general
algorithms for segmentation and registration. To support our program, we will develop modular
software frameworks in which it is easy to compare alternative methods within the context of an end-toend system, and in which it is easy to interconnect suites of modules to build complete systems. Because
we see such integrated systems as central to the development of IGT systems, we will also explore
concepts for managing software development and assuring quality, especially within a multidisciplinary
multi-location collaboration.
Initially, we will use image guided prostate brachytherapy as our testbed. We will then expand into
minimally invasive liver procedures and, finally, towards the end of the duration of the grant period, we
will work on image guided breast surgery. This approach will present us with increasing difficulties in
compensating for nonrigid and semi-rigid motion. Both the efforts in the prostate and in the liver are
ongoing clinical projects with a constant stream of clinical cases. The breast project is in the early stages
of clinical application.
Our ultimate goal is to create the computational infrastructure and associated suite of methods to
support a broad range of image guided therapy procedures, and to evaluate the impact of such IGT
interventions in the delivery of surgical care.
Page 74
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
B. Background and Significance
B.1. Significance
Images are rich in information that can be used for diagnosis and for subsequent therapeutic
interventions. Our goal is to develop tools that leverage the application of image-based information to
the tightly coupled processes of diagnosis and therapy. To appreciate the potential impact of IGT
systems, we consider the numerous ways in which images influence diagnosis and therapy. IGT systems
use preoperatively acquired images to create anatomical models, which provide localization, targeting
and visualization of the three-dimensional (3D) anatomy. These models support preoperative planning to
define and optimize access strategies as well as the simulation of planned intervention. When registered
to the patient, these models connect image coordinates with the actual position defined by an
instrument’s location in the surgical field, thereby enabling a surgeon to navigate and execute
procedures with full knowledge of the surrounding anatomy.
Used in these ways, image based models can support a variety of medical applications. For
diagnosis, the primary objective is the detection, localization and identification of a potential
abnormality. For therapy, the primary objective is localization, which includes not only the exact
anatomic position and spatial extent of an already known target, but also the delineation of surrounding
anatomy and the comprehension of important anatomic relationships. The depiction of anatomic
structures not adjacent to a target may be important if they are located within the operational volume or
reside along one of the potential access routes to the tumor target. Image-guidance supports both
objectives by providing 3D representations of the target and operational volume, including tumor
margins, tissue identification, and nearby structural content.
Localization or target definition should incorporate all the essential morphologic, anatomic and
physiologic properties of the target. These are required for planning and executing interventional and
surgical procedures, especially for optimizing targeting and achieving complete removal or ablation.
Target definition can improve not only the detection of tumors but also the effectiveness of surgical
therapies. Image tools are essential at this stage.
The localization of a tumor and its delineation in 3D defines the target of therapy. The next step is
the selection of potential access routes or trajectories, wherein the operator must choose preferential
options from a multitude of alternative paths to the lesion. In the case of biopsy a single trajectory
should be chosen. During surgical procedures, or more complex percutaneous interventions, the decision
usually involves multiple conceivable trajectories. Again, detailed image information is critical in
guiding decisions at this stage.
Given analyzed preoperative image data, the spatial information it represents should be tightly linked
to the patient. This registration process determines the transformation of image-based coordinates into
the patient’s frame of reference, thus allowing targeting and execution of the actual procedure to occur
with optimal information available to the surgeon. Major progress has been made by moving from frame
based stereotactic systems to the current CT or MRI-based computerized frameless stereotactic systems.
These new systems use fiducial markers or anatomic landmarks to establish correspondence between the
image-space and the patient's anatomy. Both fiducial markers and surgical instruments can be tracked
using a variety of sensors. These tracking methods are used not only to relate the positions of markers or
instruments between two corresponding frames of reference but also for interactive display. These
navigational systems present images with orientation and location defined by the position of the tracked
device, which can guide the surgeons to the target lesions with relatively good accuracy unless the
Page 75
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
position of the targeted structure and/or the surrounding anatomy has changed significantly during the
procedure.
In some cases, shifts and deformations of soft tissues that occur during surgery, due to mechanical
factors, physiologic motion, swelling or hemorrhage, may displace organs or their tissue components to
such a degree that preoperatively acquired image-based 3D models cannot register or fuse with the
patient’s actual anatomy. In this situation either partial correction of the 3D model or full volumetric
image update is necessary. Even limited revision of the original images requires some clues about the
changes taking place during surgery. Some useful information can be obtained from prior knowledge
and experience-based models of deformations. However, only intraoperative imaging can provide real
information. This updated positional data then can be used to modify the original volumetric images
using elastic warping algorithms or other more elaborate computer based methods. Thus, images are a
key component in maintaining surgical integrity.
The ultimate solution for accurate image-guided targeting is real-time intraoperative imaging or at
least frequent updating of the volumetric images during procedures. This results in targeting methods
that can continuously detect changes of the position of various tissue components and locate the targets
and their environs in order to define trajectories to the lesion. Intraoperative imaging may use multiple
projections or image planes to get 3D data, or cross-sectional imaging methods like CT or MRI can be
used to encompass the entire 3D volume without the need of real-time interactive imaging.
Our main goal is to facilitate full utilization of anatomic and functional information accessible by
current medical imaging methods in image-guided therapy. By providing the surgeon with easy access
to this multi-modal information, which is registered to the anatomy of the patient, we will improve the
safety and efficiency of surgical procedures. In order to accomplish this desirable goal we must develop
methods that automatically convert medical images into patient-specific models. We will apply our
segmentation techniques to a range of multi-modal acquisitions, which we then register to a common
coordinate frame. This augmented patient model can then be used for planning and simulation, or can be
further registered to the actual patient, to support surgical visualization, and navigation.
It is our belief that this registered information may help to dramatically change surgical procedures
by enabling the surgeon to precisely identify and avoid critical structures, which are either hidden below
the currently exposed surface or indistinguishable to the human eye compared to surrounding structures.
This multi-modal information will provide the means to accurately locate pathological tissue and
facilitate trajectory optimization. It will support minimally invasive procedures, which take less time,
involve less removal of tissue, and have fewer risks of side effects.
To achieve our goal of IGT, we must do more than just develop robust algorithms for segmenting,
registering and visualizing anatomical and functional reconstructions of patient anatomy. We must also
integrate these computational tools into complete systems that are subjected to extensive use in real
surgical settings. Furthermore, we must evaluate the efficacy of the individual components and the
complete systems in improving therapeutic delivery. The feedback obtained from surgical utilization of
IGT systems is essential to developing practical biomedical toolkits for leveraging information in
medical imagery. The Surgical Planning Laboratory is well placed for achieving this dual goal of novel
algorithmic development and clinical application and evaluation. We currently provide pre- and intraoperative visualization services to our neurosurgical colleagues on a regular basis, and we collaborate
closely with users of the interventional MRI unit at Brigham and Women's Hospital. Both surgical
settings provide immediate and valuable feedback to the designers of the IGT components and systems.
Page 76
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
B.2. Background
As outlined by our Specific Aims, we consider relevant prior work in each of our proposed work’s
constituent parts.
B.2.1 Algorithms
Segmentation converts medical images into anatomically or functionally distinctive structures, which
are of more utility to the surgeon than the individual image slices. The process also identifies surface
boundaries of connected tissue types, enabling visualization of major anatomical structures. By creating
automated tools for segmentation, we enable the transformation of raw image data into structures that
directly relate to the patient's anatomy, and thus to the surgical procedure. Common approaches to
segmentation include manual segmentation, pattern recognition (statistical classification) based
segmentation, alignment based segmentation and curvature flow based segmentation. The earliest
approaches to image segmentation were achieved through intensive user interaction. Although
considerable progress has been made in the development of automated segmentation algorithms, the lack
of robust techniques still remains the most significant obstacle to the widespread use of medical image
analysis for diagnosis and treatment planning. An extensive review of MRI segmentation approaches
can be found in [Collins92], and from our group, in [Kikinis97] and [Warfield98d].
a) Segementation
(1) Adaptive Filtering
Because image intensities associated with different tissues may be difficult to discriminate in the
presence of sensor noise, several authors have applied signal-processing methods to enhance the image
intensities (feature enhancement) [Jain89]. Several methods of adaptive filtering have been proposed
and applied for image enhancement [Knutson83], [Perona90], [Restrepo88]. Initial experiments with
diffusion-based algorithms for image analysis were motivated by the aperture problem [Koenderink84],
[Witkin83], [Lindeberg90]. More advanced methods included nonlinear diffusion and geometry-driven
diffusion [Nordström90]. Level Set methods are increasingly finding application in medical image
processing [Osher88], [Sethian89], [Sethian92]. They provide powerful and general-purpose means of
evolving surface models in three dimensions. When coupled to image data, they can be used to segment
or identify structures having certain properties. These methods were used by [Zeng98] to find the sulcogyral structure of the human cortex from MRI. Snakes, or active contours, are a common computer
vision tool, and have been used for edge and curve detection, segmentation, shape modeling, and visual
tracking [Blake93]. In general, partial differential equation methods that couple to image features
localized by curvature driven evolution are under active investigation [Morel95].
(2) Intensity Based Classification
The use of multi-channel statistical intensity classifiers in the medical domain was pioneered by
Vannier et al. [Vannier85] and later used and extended by many others. This class of methods models
each tissue type as a distribution of intensity values in some feature space, and uses variants of a nearestneighbor rule to classify voxels based on recorded intensity. The tissue model distributions may be
acquired from labeled training data, or provided by a priori models. Variations of this method include
the use of ``homomorphic" methods, [Axel87], [Lim89] to correct for slowly varying intensity artifacts,
as well as several non-homomorphic approaches [Dawant93], [Tincher93]. Several authors have
reported methods based on the use of phantoms for intensity calibration [Axel87], [Gohagan87]. Several
authors, including [Kapouleas94] and [Kohn91] have examined the use of statistical neighborhood
models for segmentation. The use of Markov Random Field neighborhood models to capture local
variations in voxel classification has also received attention [Held96], [Geman84]. Additional
Page 77
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
anatomical knowledge can be factored into these methods by modifying the classification of a pixel,
based on registration of the image volume to an atlas of empirically determined tissue class priors
[Kamber95], [Zijdenbos98].
(3) Segemntation by Alignment
More recent approaches to segmenting anatomical structures consider taking a template data set and
comparing it with patient data by finding a transformation that aligns the reference data set with the
patient data. In this way, one can either apply transformed labels from the template to define tissue
boundaries, or one can use the transformed template to set expectations for the statistical classification
of the patient data. This approach applys anatomical context to aid in the segmentation process, though it
clearly depends on the existence of an accurate, detailed atlas, and the ability to warp that atlas to the
new image data. This registration of an atlas has been accomplished through manual correspondence
[Evans91], [Bookstein92], semi-automated correspondence detection [Collins92], [Meyer96], and
automated correspondence [Dengler88], [Bajcsy89], [MacDonald94], [Moshfeghi94], [Collins94],
[Thompson96], [Szeliski93], [Christensen94], [Bro-Nielsen96], [Miller93]. Several different schemes
have been successfully applied for representing the high-order nonlinear transform necessary to align a
reference template with a patient, allowing interesting characterizations of anatomical variability
[Thompson96], [Thompson97], [Haller97], [Gee92].
(4) PDE’s and Snakes
Grenander introduced deformable image models in his classic work on pattern theory [Grenander76].
The formulation of pattern understanding has been made in terms of Bayesian statistics. (Utilizing
concepts from information theory, in particular the notion of “minimum description length”, allows the
derivation of an equivalent formulation.) The fundamental idea of incorporating prior knowledge on the
space of possible images in the form of a template on some domain, and the assumption that the class of
all possible true images is formed by a composition of that template with continuous mappings from the
domain into itself has had a major effect on various computer vision and image processing algorithms,
in particular algorithms designed for the problem of segmentation. Indeed, the philosophy underpinning
the well-known Mumford-Shah functional which gives a rigorous variational approach for segmentation
[Mumford89] is strongly based on Grenander’s work. Here one wants to find the best match to the
deformed image; “best” is defined in terms of the given functional. This in turn has motivated much of
the research in the PDE based approaches for segmentation [Morel95]. Finally, deformable templates
can powerfully model the variability of observed imagery, and have been to a large extent rigorously
justified; see [Amit91], [Grenander98] and the references therein. In our proposed work on
segmentation, this constellation of concepts based on pattern theory certainly will play an important role
when we consider the organization of our methodologies for segmentation and feature extraction.
(5) Shape Description and Modeling
Shape theory is an essential problem for many computer vision tasks [Koenderink90]. Indeed, the
problem of shape may be considered the principal bottleneck between high and low level vision. Much
of the work we will discuss below is connected either implicitly or explicitly to modeling and describing
shapes. It is important to note that while there is a sense in which the meaning of shape is intuitively
understood, a formal definition has been elusive. Many early visual processes contribute to shape
including edges, texture, color, and shading. There have been many methods proposed for shape
classification including shape feature descriptions such as area, eccentricity, centroid, compactness,
shape moments, as well as algebraic and differential invariants, and Fourier descriptors, to mention a
few. We will describe here some of the approaches that have influenced our work, as well as the
proposed research in biomedical imaging to be considered below.
Page 78
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
The skeleton or medial axis has become a standard shape descriptor since being introduced in
[Blum73]. (The symmetry set of a planar curve is defined as the set of points equidistant from at least
two different points on the given curve provided the distances are local extrema; the skeleton is a subset
of this set.) Recently, using statistical analysis and the notion of fixed topology skeletons, Golland et al.
[Golland99] have employed this methodology as the basis of a shape analysis study on corpus callosum
data for schizophrenia patients. We should note that because of the sensitivity of the skeleton to noise,
researchers have been forced to consider a number of possible solutions including pruning, multiscale
and hierarchical descriptions of skeletons, and curve evolution methods; see [Bruce86], [Kimia95],
[Ogniewicz93], and the references therein. The fixed topology skeleton is another approach for treating
the noise problem.
Deformation techniques based on level sets have also proved useful for shape modeling; see
[Caselles93] and [Malladi95]. One may also use partial differential equations implemented on
triangulated surface representations for surface deformations in order to develop shape models and
metrics as in [Angenent99a]. This latter approach will be described in greater detail below.
b) Registration
Registration deals with the process of aligning data sets either to other data sets or to the actual
patient. Such alignment may be a simple matter of finding a rigid transformation between a small set of
landmarks, or may involve elastic warping of complete data sets to one another. The general framework
for registration is: given two data sets, find the mapping or transformation from one set to the other,
which optimizes some criteria of similarity between the two data sets. Thus key factors include the class
of transformation solved for, the types of data used to define the correspondence between data sets, and
the methods use to find the optimal transformation.
(1) Rigid Registration
Needs to be done by Simon/Noby
(2) Non-Rigid Registration
(a) Automation
Registration techniques show considerable variation in the amount of automation available in the
method. An example at the manual end of the spectrum is the work of Bookstein [Bookstein91] on the
morphometric analysis of biological shape data as captured in manually established landmarks. This
work also uses mathematical models of deformation (the thin-plate spline) and provides carefully
considered statistical methodologies for posing and testing shape hypotheses. An intermediate point on
the automation scheme is demonstrated by the live wire systems [Barrett96] for manually guided
segmentation, and the snake methods [Kass88] in which an operator specifies an approximate starting
contour for a segmentation. In these methods, the user provides some initial guidance to steer the
method, which then automatically optimizes the registration. At the other end of the spectrum are the
fully automatic methods, exemplified by several methods of rigid registration [Wells95], [Hill94],
[Collignon95], [Studholme95a], [Woods93], [Thirion95], and some non-rigid methods [Bajcsy89],
[Christensen94].
(b) Deformation Model
Registration techniques also vary according to the deformation model that is employed or allowed for
shape variations. Among the simplest methods are those that solve for a rigid registration of the data
sets. An intermediate approach is to use a mathematical model that has limited degrees of freedom
(DOF). Examples of this sort include global linear or affine deformation models. Additional DOF may
Page 79
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
be obtained with piecewise linear or affine methods by allowing a significant number of independent
compartments via the use of a mesh of control points, or vertices. This approach may also be used with
spline methods, in particular thin-plate splines [Bookstein91]. The thin-plate spline method has been
used in a mutual information (MI) based registration scheme [Kim96]. Some brain registration methods
have made use of deformation models that have, ideally, unlimited degrees of freedom. Such models are
often formulated in terms of continuum mechanical models, e.g. liquid or elastic solid models. Bajcsy
and Kovacic [Bajcsy89] represents early work on non-rigid registration that uses a deformable 3D grid
model We employ a similar method, due to Dengler [Dengler88] in our own template-driven
segmentation work [Warfield95]. In later work, Christiansen et al. [Christensen94] have described a
method of brain registration across individuals that employs a fluid transport model. Thirion [Thirion95]
has reported similarly impressive looking results with faster running times using a method described in
terms of ``demons'' that locally deform the model to push it into a “mold”.
(c) Coupling to Data
Another important aspect of registration systems is the method that is used to couple the deformable
object to the image data. Some work has made use of sum squared intensity differences (correlation like
measures), but this approach can have difficulties with differing modalities, and missing or extra
structures in one of the data sets. Van den Elsen [Van den Elsen 93] describes an approach where CT
data is intensity transformed to make it more like MRI, and then correlation is used.
Several researchers have investigated the use of joint entropy [Studholme95b] [Collignon95], and
found that it was not a robust measure of registration, due to difficulties associated with partial overlap
of the data. Collignon et al. [Colignon95] and Studholme et al. [Studholme95a] found registration based
on mutual information to be an attractive approach, with Collignon et al. [Collignol95] describing the
use of Powell's optimization method. Our own work in intensity-based medical applications of rigid
registration [Wells96] has used mutual information as a measure of agreement, and this method of data
coupling has also been used in non-rigid registration [Kim96]. We have recently investigated a new
method of data coupling that may prove more robust than MI in some applications [Leventon98], and
we intend to characterize this approach for the applications described here.
(d) Accommodation of Topological Changes
Some registration problems, particularly atlas registration, must accommodate differences in
topology, which requires interpolation of information between known key points. The mathematics of
this process has been developed in the field of computer vision, including early work by Grimson
[Grimson84]. Terzopoulos [Terzopoulos86] has demonstrated a surface reconstruction method that is
able to generate smooth surface representations between discontinuities. Another approach was
developed and used successfully for tracking moving objects with an optical flow technique by Dengler
[Dengler88]. A controlled continuity function was introduced to allow local control of the degree of
regularization applied by the membrane through the use of a scaling of the membrane deformation
energy. Each of these methods can be converted to the medical registration problem, by interpolating
transformation fields based on information obtained at key landmarks or other features.
B.2.2 Software Engineering
The concept of the web browser is a paradigm that literally hundred of millions of people worldwide
are learning. Its power lies in both its universality and its simplicity. By relying on the same kinds of
networking and communications standards that make the world wide web possible, we propose to tailor
this paradigm in a way that is appropriate to the dissemination and understanding of multimodality and
multi-representational biomedical data in a medical environment. This part of the proposed work builds
on the network awareness we will build into the visualization pipeline. Once the visualization pipeline
Page 80
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
can transparently read data off of the network and output images and geometry in a standardized form, it
can be controlled from a web-based interface and thus enhance the user's ability to understand the data
available. To the user, the web interface becomes a portal or clearinghouse of all the information known
about a patient, expressed in any of the ways that visualization tools can provide.
Based on this model of data interaction, we will move forward and implement dynamic interaction
tools within the web environment. These tools will permit, for instance, a user to view not just a static
image of a patient's anatomy, but rather an interactive, three-dimensional model. These tools will be
built using standards such as VRML [VRML97] X3D [X3D99] and Java3D. The tools that are available
to a user will depend on the user's identity and the capability and location of the computer upon which
the data is viewed.
B.2.3 Integrated Systems
The software interface, vtk [Schroeder98], is the primary substrate for the modularized visualization
architecture upon which the proposed work is based. Modular tools for data visualization have seen a
progression of development since the introduction of AVS in 1989 [Upson89]. Existing modular tools
include the commercial systems IBM Data Explorer, Wavefront Data Visualizer, Iris Explorer and AVS.
Each of these tools allows a user to construct a data visualization application using a toolbox of
algorithms. The user needs only a high-level idea of the implementation of each algorithm: the toolbox
exposes only algorithm parameters and data inputs and outputs for each module. vtk's development
methodology shares the toolbox concept with these previous works. Modules implementing
visualization algorithms are networked together using a scripting or programming language to form a
complete visualization application. vtk is distinct from these other works in several ways. First, vtkbased visualization pipelines are more naturally embedded in applications since elements are represented
and connected programmatically rather than visually. Second, the entire internal structure of vtk is
exposed to the programmer developing modules and algorithms, providing a great deal of flexibility in
handling data and providing functionality. Third, the programming source code for vtk is open and
freely available, providing insight into its inner workings and allowing peer review of its methods.
Finally, vtk is a vital, evolving toolkit with a diverse and active developer community.
C. Preliminary Results
We would like to continue our ongoing development and implementation of bioengineering methods
for IGT. When our Image Guided Therapy Program at the Brigham and Women's Hospital was
established, the field was in its infancy and intraoperative image-guidance was only a great but
unsubstantiated promise. Today most of the intraoperative applications we have developed are in the
routine clinical stage, and new applications have been tested in our Hospital and elsewhere. Based upon
our preliminary experience we are now convinced that intraoperative guidance may allow surgical
procedures to be performed more precisely and with less morbidity than with conventional techniques.
Intraoperative image-guidance that relies on preoperatively acquired images may provide important
information to the surgical procedures but it should be complemented by real-time intraoperative
imaging or with frequent image updates. The preoperative multi-modality information should be
complementary to the near real time information [Jolesz96]. In our Program we have been combining
advances in navigation, 3D multi-modality image representation and intraoperative MRI in order to
achieve an integrated image guidance system [Jolesz98]. We have acquired considerable experience in
development components and integrated systems for IGT. Although this has been primarily in the
domain of neurosurgery, we have also developed computational methods for other domains, such as
Page 81
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
spine, knee, shoulder, liver, and breast. Below we briefly describe our primary results that directly relate
to the present proposal.
C.1. Algorithms
C.1.1 Segmentation
We have an extensive background in the segmentation of medical images, including methods based
on statistical distributions of intensities, methods that apply local geometric information, and methods
that use anatomical atlases and atlas-based templates to drive segmentation.
a) Adaptive Filtering
An alternative approach to segmentation focuses first on processing the image data, in order to
enhance tissue differences. Once such feature enhancement has been performed, identification of
distinctive tissue types is often simplified and can be performed by existing methods.
We have investigated using local 3D structure for segmentation [Westin97], [Westin98]. A tensor
descriptor is estimated for each neighborhood, for each voxel in the data set. The shape of the tensors
describe locally the structure of the neighborhood in terms of how much it is like a plane, a line, and a
sphere. We apply this to segmentation of bone from Computer Tomography data (CT). We show that by
using additional local structure information we can significantly reduce the degree of sampling artifacts.
(1) Bone enhancement (Media Paper)
(2) Vessel enhancement
b) Intensity based Classification
Developing automated signal intensity based segmentation has been a major focus of our work for
several years. Early work includes [Cline90], [Kikinis92], [Wells96a]. We demonstrated the use of
classification in combination with manually determined connectivity constraints for the segmentation of
normal and some abnormal tissues from the brain. It also described a method for the use of surface
rendering for visualization of segmentation results.
In subsequent work, we performed quantitative analysis of brain and CSF spaces with MRI
[Kikinis92]. In that paper we describe a supervised statistical classification approach to segmentation,
and details the accuracy and reproducibility of the segmentation. The paper demonstrates our approach
to validation with manual outlining by experts. Five experienced raters used both the automated
segmentation procedures, as well as manually outlined regions on the same slice. We compared how
closely the raters agreed within a single method (automated or manual), and how closely the same tissue
was classified. In all cases (i.e., classifying intracranial cavity (ICC), gray matter, white matter,
cerebrospinal fluid (CSF)), automated segmentation procedures were superior to manual procedures.
Agreement among raters also correlated with the complexity of the object being measured, such that the
more complex the object’s shape, the lower the agreement.
In [Wells96a] we specified an algorithm that imroves the accuracy of MRI segmentation by explicitly
characterizing a slowly varying, non-linear gain field that is present in the images and is attributable to
inhomogeneities in the imaging equipment. This gain field is sufficient to perturb the underlying
distributions that are used in traditional intensity-based classification. We formulated the segmentation
of MRI images as a maximum likelihood estimation problem and used the Expectation-Maximization
algorithm to simultaneously estimate the class label and gain at each voxel that maximize the likelihood
of the observed signal. Specifically, we modeled the observed MRI signal as a product of the true signal
generated by the underlying anatomy and the non-linear gain artifact. Using this assumption, we
Page 82
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
developed an iterative, supervised, Expectation-Maximization style segmentation algorithm that treats
the underlying label classes as hidden variables and alternates between estimating those classes (E-step)
and the maximally probable gain field (M-step). The alternated estimation of these components allows
both tissue segmentation and intensity correction to be estimated without manual interaction with each
case.
Next we addressed the problem of fragmentation in segmentation that is due to the presence of
thermal noise in MRI scans. We incorporated a regularizing Markov random field (MRF) prior into our
EM-Segmentation method that captures the piecewise homogeneos nature of tissue[Held96]. While our
original EM-Segmentation algorithm assumed independence between the tissue labels at adjacent sites
(voxels) in an image, with a Markov prior, we incorporated constraints on the tissue labels at a site based
on the estimates of the labels at adjacent sites. Estimation problems involving MRFs can be solved
using a variety of deterministic and non-deterministic techniques and in this work we used a greedy
deterministic method, Iterated Conditional Modes (ICM), as our solver.
More recently, we continued our effort in denoising MRI by incorporating a Mean-Field
Approximation in the EM-Segmentation algorithm [Kapur98, Kapur99]. In this work we used a Gibbs
prior (equivalent to Markov prior) on tissue class and solved this Gibbs model using a Mean-Field (MF)
approximation in conjunction with Expectation-Maximization. In contrast to an ICM solver which
computes discrete estimates of tissue class at each iteration, a MF approximation uses continuous
estimates of the classes which is desirable because the EM-Segmentation algorithm represents tissue
class as continuous values. In [Kapur99] we showed that the denoising achieved by this method, EMMF Segmentation, are comparable to first denoising the images using an anisotropic diffusion filter
followed by EM-Segmentation. The integrated EM-MF method is thus a suitable replacement for the
more compute-intensive, pipelined method of denoising followed by segmentation.
Another synergistic mechanism for improving classification in the Expectation-Maximization
framework is to modulate the prior probability of tissue classes spatially. This is in contrast to the
previously discussed MRF priors that are spatially stationary (invariant). In [Kapur98a] we introduced
relative geometric models which are statistical descriptions of the spatial relationships between
structures. These computational models capture notions of spatial relationships such as "cartilage age
lines the bone'' or ``white matter lies inside the skin boundary'', that are likely used by experts in
segmentation of structures that exhibit poor intensity contrast in images. We used these models as
spatially varying priors on structures in conjunction with EM-segmentation.
c) Segmentation by Alignment
A complementary approach to segmentation applies anatomical information to the problem, by
registering atlases or other representations of canonical models to the data set. Our earlier work on
segmentation by registration has concentrated on several components [Warfield98d]. To generate a
digital atlas of normal tissue, we have carefully converted a high-resolution T1 weighted MR data set of
0.9375 x 0.9375 x 1.5 mm3 resolution, obtained from a prospectively studied normal volunteer, into a
detailed digital atlas [Kikinis96a]. To register this atlas to new imagery, we combine robust linear
registration algorithms for initial alignment of atlas and subject data set, followed by a fast elastic
warping algorithm for nonlinear registration of the two data sets. These methods [Dengler88],
[Wells96b], [Warfield98b] are described in detail in the registration section. Finally, we have developed
and validated an image analysis system that uses segmentation by registration to identify deep gray
Page 83
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
matter structures and the white matter in the brain MRI scans of patients with multiple sclerosis. This
was shown to allow improved segmentation of white matter lesions [Warfield95] over previous
methods.
We have proposed a non-Bayesian approach to modifying classification behavior, through the use of
a template of normal anatomy [Warfield98c]. This work differs from the conventional segmentation by
alignment approaches in that the template is used not to directly project boundaries onto the patient but
instead to moderate the classification behavior spatially. In particular, our template moderated
classification approach does not require the topology of the data set to be segmented to be identical to
that of the anatomical model. This is particularly important when dealing with scans that exhibit gross
pathology.
Segmentation and validation experiments were carried out for problems involving the quantification
of normal anatomy (MRI of brains of neonates, MRI of knee cartilage of normal volunteers
[Warfield98a]) and pathology of various types (MRI of patients with multiple sclerosis, MRI of patients
with brain tumors, MRI of patients with damaged knee cartilage). In each case, the TMC algorithm
provided a better segmentation than statistical classification or segmentation by registration alone
[Warfield98c].
(1) Simons MICCAI
(2) Simons Media
d) PDE’s and Snakes
In [Kichenassamy96], we have proposed a novel deformable contour model for use in segmentation.
We define a new Riemannian metric in the plane tailored to the given image, and then compute the
corresponding gradient flow. This leads to some new snake models that efficiently attract the given
active contour to the features of interest (which basically lie at the bottom of a potential well). We have
generalized this to 3D active surface models [Kichenassamy96], [Tannenbaum96]. Our 3D active
contour evolvers can be used for image segmentation, shape modeling, and edge detection based on both
snakes (inward deformations) and bubbles (outward deformations) [Yezzi97]. The implementation of
this deformable contour model using the level-set approach makes it ideal for applications in which
initial estimates of object shape can be evolved to capture detailed structure from local image
information about surface boundaries. It is important to note that various types of structures can be
naturally detected using these methods depending on the choice of stopping (conformal) function. Thus
in addition to intensity based edges, one can segment such vectored-valued features as color and texture.
Moreover, statistical and adaptive based schemes such as those outlined above can also be integrated
into the stopping criterion which can be essential for nonstationary data.
Besides being utilized to evolve hypersurfaces (e.g., curves in the plane or surfaces in space), level
set techniques can be extended to evolve space curves. This is important in detecting thin vessels, and
has recently been successfully applied in [Lorigo99] to segment vessels in MR angiography.
As well, [Angenent98b] have applied the level-set methodology to identifying the cerebral cortex in
brain MRI and mapping it onto a sphere.
Page 84
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
(1) Allens paper
(2) Andy Tsai MICCAI (include picture)
(3) Lianas IPMI
C.1.2 Registration
At the Surgical Planning Laboratory we have considerable experience, both in the development, and
in the application of rigid registration to image-guided surgery. This activity has concerned patient to
MRI registration for surgical guidance as well as the fusion of pre-operative data.
The Surgical Planning Laboratory has considerable experience in the application of rigid registration
to problems of pre-surgical planning and intra-operative guidance in the conventional operating room,
and in conjunction with intra-operative MRI.
a) MI algorithm
Since 1994 the Surgical Planning Laboratory has been providing a service of fusion or registration of
volumetric images for the pre-operative preparation of coordinated multi-modality data sets. This data is
used for surgical planning and guidance, as mentioned above. Most of our work in this area has used the
method of Maximization of Mutual Information [Wells96b]. We were the original developers of this
approach, which has since been adopted by many sites around the world. It has also been extensively
tested, with favorable results, in a NIH-funded trial of algorithms for retrospective registration of multimodality data sets [West97].
(1) Sandy/slicer/gering
b) Interactive Registration
Surgery for intracerebral tumors can be significantly improved by image guidance to define the
margins of tumor, to avoid eloquent areas of brain, and to judge the extent of resection. At Brigham and
Women's Hospital we have used two methods of imaging for neurosurgical resection: three dimensional
reconstruction of preoperatively acquired images with navigation within the surgical space thus created;
and the intraoperative MRI scanner which allows near real-time updating of images during surgery.
We have been providing neuronavigation capabilities to our collaborators in the neurosurgery
department since 1993 [Grimson96], [Grimson98], [Chabrerie98]. This evolving project has been used
to provide image-based guidance in more than 300 neurosurgical procedures. The basic capability
involves registration of the patient in the operating room to pre-operative MRI, and thereby to a suite of
previously prepared and fused multi-modality image data, which may include CT, SPECT, fMRI or MR
angiography. This registration is accomplished by measuring the skin surface profile, either with a laser
scanner [Grimson96] or with a manually operated, instrumented probe [Grimson98].
We have developed and tested a preliminary method to fuse the multi-modality (pre- and intraoperative) images for image-guided surgery. The preliminary work included the engineering setting of
high-performance computing, the use of the hospital network and the integration of the interactive
guidance system of the open-configuration MRI scanner. We have concluded that highly accurate and
complex surgical procedures are possible using an interactive 3-D graphics display in which we can
integrate multiple modalities (MRI [T1, T2, MRA, fMRI], CT, SPECT, and intraoperative MRI data). A
manuscript describing the preliminary results of this system implementation is in preparation. It is
already obvious that using this method surgeons and neuroradiologists can localize a trajectory path that
is minimally invasive to critical anatomical structures such as vessels and critical cortical functional
area.
Page 85
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
(1) Noby/Andreas prostate work
c) Registration for TDS
(1) Rigid
(2) Elastic Dengler
This part of the proposal is based, in part, on information-theoretic work in image registration and
object recognition [Wells96]. This registration method has demonstrated robustness with respect to the
fusion of different image modalities. The method has been further developed for medical registration
applications, among them MRI - CT, MRI - MRI angio, and MRI - PET [Wells95]. A significant
advantage of the method is that it operates on the image intensities directly; no segmentation or preprocessing is needed.
To date, the main applications of non-rigid registration in our lab have been the automatic
segmentation of MS lesions in MRI [Warfield96], and the automated segmentation of tumor and brain
tissue from MRI [CITE KAUS MICCAI99, RADIOLOGY PAPER THAT HAS BEEN SUBMITTED].
The difficulty is that in many data sets, there is ambiguity in signal intensity between healthy tissue and
diseased tissue. To get around this ambiguity, we have used elastic warping of the individual's data set to
a labeled atlas for the purpose of estimating the anatomical boundaries between the individual's gray and
white matter.
For elastic matching, we are using an approach that is similar in concept to the work reported by
Bajcsy and Kovacic [Bajcsy89], Collins and Evans [Collins92]. However, our implementation
[Warfield98c] uses several different algorithmic improvements to speed up the processing, a multiresolution approach with fast local similarity measurement, and a simplified regularization model for the
elastic membrane [Dengler88]. We have applied this approach to a range of segmentation tasks in
different areas of the body and have demonstrated that this approach [CITE WARFIELD PHD THESIS,
1999 MedIA paper] improves the specificity of the segmentation compared to the results achieved using
statistical intensity-based segmentation alone, while maintaining excellent reproducibility. In particular,
in multiple sclerosis segmentation, we were able to overcome overlap of signal intensities between white
matter lesions and gray matter by automatically defining the region of the white matter.
(3) Mathieu Ferrant
We have developed expertise in the area of nonlinear registration through our work in inter-subject
registration. Our early work investigated the generalization of algorithms from 2D computer vision
motion capture work [CITE DENGLER 1988], which used an optical flow based similarity measure
coupled with an elastic membrane model for regularization. Since that time we have enhanced this
approach with optimized implementations that are sufficiently fast to make routine clinical use practical,
and we have expanded our modelling capability to allow us to better capture shape differences between
subjects [CITE FERRANT, HATA PhD THESIS].
The clinical focus of this application deals with areas of the body for which non-rigid tissue
deformations are an essential characteristic. Our most significant experience has been Kapur's
experiments in registration of breast MRI [in preparation, reference URL showing mpeg visualization],
Hata's work in non-rigid registration in the brain [Hata98b, HATA JCAT 99, HATA MICCAI99], and
Ferrant’s work in non-rigid registration in the skeletal musculature [FERRANT MICCAI 99].
In a later section, we will outline a method of registration based on the use of brachytherapy seeds as
landmarks for registration. This type of matching and registration problem has been extensively studied
by Grimson (see e.g. [Grimson84], [Grimson96] and [Grimson98]), and relies on a common technique
Page 86
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
in computer vision, called interpretation tree search. Such registration methods can easily match
landmarks, and then serve as a basis for interpolating smooth transformations between those landmarks.
(4) Shape Representation, Modeling, and Deformation
We have been working on several approaches to shape, especially as applied to brain imaging. First of
all, the skeleton introduced by [Blum73] more than a quarter of a century ago into vision is still
influencing our research work. The key problem (as always) is the sensitivity of this shape descriptor to
noise. To account for this in some recent work Golland et al. [Golland99], introduce the notion of a
“fixed topology skeleton” in which the structure of the skeleton graph is fixed, and then optimize for the
accuracy of the original shape representation over all the skeletons with that fixed structure. This is
useful for the shape analysis of anatomical structures, since the general shape of the object is already
known, the possible deformations are small and do not change the overall shape.
In [Martin98], a framework is proposed for analyzing the shape deformation of structures within the
human brain. The total shape deformation is decomposed into analytic modes of variation obtained from
finite elements, and statistical modes of variation from sample data. The goal is to separate important
from unimportant shape variation in a given class of objects. Modal analysis is applied to the physical
modeling, and principal component analysis to the experimental observations. Other work on physically
based shape representations may be found in [Bookstein89] and [Pentland 91].
We have also been considering another deformation approach based on very different principles.
Deformations of the cortical surface, and in particular, flattening the brain surface have many uses
including functional MRI. Indeed, since it is important to visualize neural activity within the sulci,
flattened representations of the cortex have become increasing widespread. In [Angenent99a], we have
applied a partial differential equation (PDE) approach to identifying the cerebral cortex in brain MRI
and mapping it onto the sphere or the plane. Our method produces a one-to-one flattening map, which
preserves angles, and therefore the local geometry of the surface being deformed. Such a mapping is
called conformal. A similar technique may be employed for visualizing 3D colon CT data for
applications in virtual colonoscopy [Haker99a]. Consequently, as an alternative to the colon “flythrough” one may flatten the entire colon at once. The implementation of the deformation is based on
finite elements, and employs a triangulated representation of the segmented surface. The technique also
allows one to map one surface onto another (assuming that there are diffeomorphic). Thus our surface
deformation approach may be useful for registrations as well. There is also an interesting topological
approach to colon “fly-through” in [Wang98].
C.2. Software Engineering and IGT Systems
We have already developed, installed and evaluated several integrated systems for IGT. These
systems include clinical systems that integrate components into a coordinated system for use in a
particular clinical domain, and general-purpose integration frameworks that support comparison and
unification of alternative modules.
C.2.1 VTK
The visualization systems architecture upon which much of our preliminary and proposed work is
based is known as the Visualization Tool Kit (VTK). VTK is the product of two previous visualization
system designs: the General Electric Research Workstation and lymb. The Research Workstation
program was developed through collaboration between GE, Brigham and Women's Hospital, and MIT.
This software package provides a basic framework for image processing, data manipulation and editing,
manual and semiautomatic segmentation, data acquisition and conversion, and other medical imaging
tasks. Although the system consists of a small collection of programs and a monolithic user interface,
Page 87
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
Research Workstation allows new image processing modules to be added to the system. Designed as a
value-added package for users of GEs MRI scanners, the Research Workstation continues to be used
both by us and by other research sites.
The usefulness of the module features of the Research Workstation, combined with the flexibility of
user interface paradigms of visualization systems such as AVS, inspired the next internally-developed
visualization system, Lymb. Lymb itself is monolithic: all of the functionality of the system is built into
a single program. However, different process objects can be connected in pipeline fashion to produce
complex visualization networks. These process objects can be manipulated using a high-level scripting
language, improving the speed with which visualization tools can be constructed, tested, and used.
In addition to the image processing tools provided in the Research Workstation system, Lymb uses
the power of underlying three-dimensional graphics software and hardware to visualize 3D structures
efficiently. User interfaces that allow interactive manipulation of visualization parameters and threedimensional objects can also be constructed using Lymb's scripting language. Although Lymb was not
released to the general visualization community, it was the basis of several successful visualization
projects, including virtual endoscopy.
VTK incorporates the extensibility and interactivity of the Research Workstation, along with the
visualization pipeline for interactive 3D graphics and image processing from Lymb. Based on
experience from these previous systems and the state of the art in visualization, VTK was designed from
the ground up to be completely module-based, rather than monolithic. This decision means that new
modules can be easily added to VTK to provide additional functionality for data acquisition, conversion,
manipulation, and display. From the application standpoint, VTK can be used to build entire user
applications (such as the 3D Slicer described [IN SECTION ???? ]), or alternatively as an internal
component in an independent application package.
The benefits of having source code openly available for review and extension were clear even in such
internal efforts as the Research Workstation and Lymb. The distribution model for VTK is even more
flexible: the source code to VTK is freely available for inspection, use and modification. This policy
improves the quality of the software, simplifies the collaboration and exchange process, and widens both
the software's user base and its capabilities. For these reasons and for the capability that it already
provides, VTK is the basis of many of the existing and proposed tools for visualization described in this
grant proposal.
Currently, the output of VTK modules is most commonly a two-dimensional or three-dimensional
image within a custom graphical user interface. It is currently possible to produce images using VTK
that can be displayed within a web browser such as Microsoft's Internet Explorer or Netscape
Corporation's Communicator. While this process is currently done manually in a labor-intensive process,
web pages constructed in this way have proven themselves very useful for the dissemination of case-bycase documentation in medicine.
C.2.2 XPLAN
The first of our IGT systems is an image guided planning and navigation system for
neurosurgical cases, with the following components:
(1) MRI neuroscans are segmented into distinct anatomical structures (typically white matter,
gray matter, tumor, skin, bone, fat, CSF) using a combination of template moderated
classification and user guidance [CITE KAUS MICCAI99];
(2) additional scans such as MR Angiograms or functional MRI are also segmented, using
feature enhancement methods and thresholding techniques[CITE WESTIN/LORIGO];
Page 88
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
(3) these multiple segmentations are then fused into a common coordinate frame, using the
Mutual Information registration technique [CITE WELLS] - this results in an augmented
patient model;
(4) for cases involving tumors close to functional cortex (such as motor strip), Transcranial
Magnetic Stimulation is used to measure the area of the motor cortex and the motor cortex
area is projected onto the surface model of the brain to allow an integrated visualization of
both the structural and functional information;
(5) During surgery, the augmented model is registered to the patient, without the need for
fiducials or other stereotactic markers. This registered model is used in an augmented
reality visualization mode to support planning of the surgical procedure;
(6) During surgery, surgical instruments are automatically tracked and registered to the patient
surface model and volumetric data using an optical tracking system, enabling the surgeon
to visualize hidden structures, locate key landmarks, identify borders of abnormal tissue,
and verify resection of tissue.
The system has been used in over 100 neurosurgical cases to date, as documented in
[Chabrerie98a] and [Ozlen98]. Ozlen et al. describe an illustrative case where a young man suffering
from intractable epilepsy initially underwent the placement of subdural grids and strips to localize the
source of his seizures. The location of each surface electrode was recorded using trackable probes,
registered to the preoperative model, and thus displayed on the 3D model obtained from segmentation of
the MRI data. Results of the bedside neurological stimulation and monitoring were correlated with each
electrode location on the 3D model. This data was made available to the surgeon during the second
surgery, when the navigation system was used to locate the source of this patient's seizures. The system
was used for localization of the lesion, as well as to map out the language area, which lay near the
epileptic focus. The system was used to ensure that the whole lesion was removed. The patient has
remained seizure free since this procedure.
The importance of neurosurgical navigation was also demonstrated for low-grade gliomas
[Chabrerie98b]. In this paper, twenty procedures on patients with this type of tumor were performed
using 3D models obtained from segmentation together with navigational guidance. In all cases, the
navigation allowed accurate resection of the tumor. Thirteen of these cases were performed in
conjunction with intra-operative cortical mapping under local anesthesia; the neurosurgical navigator
was used to accurately establish localization of the motor cortex. These techniques provide an additional
means for surgeons to improve safety and precision, as well as to ensure complete removal of these
tumors.
C.2.3 Slicer
A more general framework for integration of visualization, registration and planning methods is
embodied in “3D Slicer”. This system can be directly connected to an intraoperative MR scanner,
allowing direct transfer of newly acquired images to this workstation. 3D Slicer uses the image
coordinates to determine the patient's location in space. Thus, this data can be used with frameless
neuronavigation systems to plan trajectory, track instruments in space, and display the position of
instruments in relation to gray-scale and 3D datasets of the patient. Visualization can speed localization
in open craniotomies and biopsies.
The 3D Slicer is a complete software package for surgical planning and guidance. Pre-operative data
sets are fused prior to surgery with a rigid registration technique employing mutual information. This
merged data representation can be visualized by reformatting up to three slices at once, and rendering
them in an interactive 3D graphics environment. Each slice can display multiple volumes blended with
Page 89
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
adjustable opacities. For example, functional information can be overlaid in color on gray scale
anatomical data.
Volumetric data can be segmented using the 3D Slicer suite of editing tools. Image processing
operations such as thresholding, connectivity, morphological analysis and free-hand drawing can be
applied to the data on a 3D or slice-by-slice basis while being visualized interactively in 3D. The output
label maps are converted to 3D surface models using the Marching Cubes algorithm. The surface models
can then be visualized in the 3D view along with the reformatted slices, and the slices can selectively
clip away portions of some models, such as the skin, to reveal other unclipped models beneath, such as a
tumor. Distances, angles, surface areas, and volumes of structures can be measured quantitatively.
C.2.4 Open MR system (MRT)
The impact of IGT systems on surgical practice can only be measured in actual surgical settings. To
evaluate reduction in surgical time, reduction in collateral damage to other tissue, improvement in
surgical planning, and overall benefit to the surgeon, one must employ full IGT systems in real settings,
and this requires careful integration of robust systems.
Imaging is an important but not the sole component of an image-guided procedure. The integration
of imaging and therapy is requisite for the control of interventional or surgical manipulations, for
optimal energy delivery during thermal ablations, and for the targeted and controlled delivery of drugs
or chemicals. This control can be accomplished only if the characteristics of the imaging systems and
the features of the therapy devices are matched and their functional properties are coordinated. These
factors all should be seriously considered before an IGT procedure is conceived, developed and
implemented. For feasibility, the limitations of the imaging systems, compatibility between the imaging
and therapy devices, and the safety features of both, should be taken into account before the initiation of
the full integration process.
The overall goal of this Project is to develop an integrated system of pre- and intraoperative image
acquisition, on-line image processing and intraoperative display that allows the utilization of all the
accessible intraoperative and preoperative information for image-guided surgery. The subject of this
Project is the integration of the software and hardware components of intraoperative sensing,
navigational tools, multi-modality imaging and intraoperative display into a single powerful image
guidance system. The goal is to provide the entire accessible image based information to assure that
intraoperative trajectory optimization and navigation can be performed in a user-friendly environment.
The vision of combining the resources of an operating room with MR imaging technology and high
performance computing is relatively new [Jolesz92], [Jolesz94], [Fried96]. It became possible to fulfill
this vision because the simultaneous combination of direct vision and imaging is possible within a
unique environment of intraoperative MRI [Shenk95]. The intraoperative MRI installed in our Hospital
incorporates both the operating room and an imaging system [Silverman96]. This application of MRI
may improve clinical outcome and reduce complication rates by reduced invasiveness. By merging MRI
with frameless stereotaxy, navigational tools and multi-modality image fusion, the combination of all
available information with image update can revolutionize minimally invasive therapy and ill result in
new treatment strategies and approaches.
The future of intraoperative MRI depends not only on the development and implementation of new
MR imaging technologies but also on the integration of computers and therapy devices into the
operating room of the future [Jolesz96]. Physicians using intraoperative MRI systems should have
access to all available image based information, which were acquired preoperatively. Appropriate
interfaces between the physicians and the integrated imaging and therapy systems are necessary to
manage this combined information. Anatomy, function and therapy induced physical changes should all
Page 90
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
be displayed simultaneously in order to achieve full control of the procedures. Today’s simpler
prototype systems may provide near real-time images or frequent volume updates for surgeries. In the
future more integrated systems will be developed which can extend the image-guidance beyond tumor
treatment into a wider range of applications including functional neurosurgery.
In this project we plan to develop a generic image guided therapy system using our extensive
experience in neurosurgical guidance [Black97] as a basis and to expand the program to other surgical
fields. The evolution of high tech integrated therapy delivery systems is closely linked with
developments in advanced imaging modalities. The operating room of the future will accommodate
various instruments, tools and devices, which are attached to the imaging systems and controlled by
image-based feedback. This will influence the further evolution of diagnostic imaging, interventional
radiology and minimally invasive surgery into a more effectual and firmly established partnership.
This combined pre-operative data set can be used to augment intraoperative MR imaging, which is
limited by a lower magnetic field and shorter imaging times than diagnostic imaging. A fast volume
scan is collected while the patient is being prepped in the operating room to create a second data
representation. Then another rigid registration using mutual information relates the pre-operative data
sets to the coordinate frame of the interventional MR scanner. As a tracked instrument is moved within
the surgical field, it is rendered in the 3D view, and the reformatted slice planes follow its position,
sweeping through the volumes. The positions of anatomical structures change as the brain swells when
the skull is opened, cerebral spinal fluid egresses, or tumor tissue is removed. Therefore, several times
during surgery, a new slab of data is collected for reformatting. The surgeon can then observe changes in
the position of anatomy on the intra-operative images, while also benefiting from the higher resolution,
higher contrast, and functional information of the pre-operative images. During biopsies and laser
ablations, 2D images are scanned in real time and rendered in the 3D display so spatial relationships can
be clearly seen.
Intra-operative MRI has been integrated into the management of neurosurgical disease at the Brigham
and Women's Hospital in the following areas to date: 1) stereotactic brain biopsy and stereotactic
placement of catheters, 3) image guided craniotomy for tumor resection, 3) image guided laser ablation
of brain tumors, 4) transsphenoidal resection of pituitary lesions, 5) image-guided spine surgery
[Silverman97], [Jolesz96], [Jolesz98], [Hsu98], [Jolesz98b], [Fried98]. We have initiated other MRIguided procedures such as laser or cryoablation of liver tumors, prostate brachytherapy, MRI-guided
lumpectomy of breast cancer, and MRI guided sinus endoscopies and MRI-guided focused ultrasound
treatment of breast tumors.
This combined pre-operative data set can be used to augment intraoperative MR imaging, which is
limited by a lower magnetic field and shorter imaging times than diagnostic imaging. A fast volume
scan is collected while the patient is being prepped in the operating room to create a second data
representation. Then another rigid registration using mutual information relates the pre-operative data
sets to the coordinate frame of the interventional MR scanner. As a tracked instrument is moved within
the surgical field, it is rendered in the 3D view, and the reformatted slice planes follow its position,
sweeping through the volumes. The positions of anatomical structures change as the brain swells when
the skull is opened, cerebral spinal fluid egresses, or tumor tissue is removed. Therefore, several times
during surgery, a new slab of data is collected for reformatting. The surgeon can then observe changes in
the position of anatomy on the intra-operative images, while also benefiting from the higher resolution,
higher contrast, and functional information of the pre-operative images. During biopsies and laser
ablations, 2D images are scanned in real time and rendered in the 3D display so spatial relationships can
be clearly seen.
Page 91
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
Intra-operative MRI has been integrated into the management of neurosurgical disease at the Brigham
and Women's Hospital in the following areas to date: 1) stereotactic brain biopsy and stereotactic
placement of catheters, 3) image guided craniotomy for tumor resection, 3) image guided laser ablation
of brain tumors, 4) transsphenoidal resection of pituitary lesions, 5) image-guided spine surgery
[Silverman97], [Jolesz96], [Jolesz98], [Hsu98], [Jolesz98b], [Fried98]. We have initiated other MRIguided procedures such as laser or cryoablation of liver tumors, prostate brachytherapy, MRI-guided
lumpectomy of breast cancer, and MRI guided sinus endoscopies and MRI-guided focused ultrasound
treatment of breast tumors.
C.3. Application Validation
C.3.1 Simon Knee Segmentation
C.3.2 Michael Kaus
C.3.3 Noby Brain Deformation
C.3.4 Thickness Mapping: From knee (Simon ISMRM) to Bladder (Julias paper)
D. Design and Methods
D.1. Design
The goal of our proposed project is the development, implementation and evaluation of biomedical
software tools and systems to support image guided therapy (IGT). Because the ultimate test of our
project will be the impact of our methods and systems in the delivery of IGT, our research plan cycles
through a series of steps, with each iteration leading to a more effective IGT system. We will use our
previous experience in individual methods and integrated systems to guide this cycle, comprised of:
Algorithm development: In the fundamental areas (segmentation, registration, visualization,
simulation), we will develop new algorithms. The design of these algorithms will be based on evaluation
of strengths and weaknesses of existing methods, analysis of the requirements of each new application
area, and incorporation of anatomical knowledge for each new area (through the use of carefully
delineated anatomical atlases).
Component evaluation: Each new algorithm will be carefully evaluated in well-controlled
situations. This will necessitate careful comparisons of the algorithm against the performance of existing
methods, under a wide range of conditions. The comparisons will be made on controlled data sets, for
which some level of ground truth is available, while maintaining realistic data conditions. For example,
in testing segmentation methods, we will compare the performance of our algorithms against manually
segmented data sets; in testing registration methods, we will compare the performance of our algorithms
on data sets where correct registration is known either a priori or through careful manual alignment.
These evaluations will enable us to measure the accuracy, robustness, repeatability and range of
operation of the algorithmic methods. Several ongoing clinical research project will provide data and
problems to drive the technology development. These include the prostate, liver and breast image
guided therapy projects.
System evaluation: In addition to evaluating each new component in isolation, we will also evaluate
each new component as part of an integrated system. This will enable us to measure the efficacy of each
new method in meeting performance requirements and specifications as imposed by the other
components of a full system. In particular, we will use the framework of a full image guided system as a
basis for comparing and contrasting new modules - how well does the new method meet the
requirements of associated modules? How well does the new method support the delivery of relevant
Page 92
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
information to the end user? Can we quantify the extent to which the system improves the delivery of
therapeutic interventions?
Surgical feedback: We will use the performance of integrated IGT system in surgical settings to
provide feedback for the development cycle. There are several key issues that such surgical evaluation
will address. What additional anatomical or functional information does the surgeon need beyond what
the current system provides? Is the accuracy of the patient model sufficient? Is the accuracy of the
registration is sufficient? To what extent are adaptable models that change to reflect modifications to the
patient needed to support surgical guidance? What impact does the image guidance system have on
surgical outcome?
This process then repeats the cycle, using feedback from the evaluation to guide the refinement of the
individual components for the next generation of methods and systems.
D.2. Methods
D.2.1 Evaluation of Individual Algorithms
To evaluate new algorithms for segmentation and registration, we will use both phantoms and real
data sets as bases. Phantoms are synthetically created, controlled data sets that support careful
examination of algorithm performance. While valuable in enabling us to isolated key factors in
algorithm development, they do not capture the full range of variability and subtlety of real data sets;
hence we will use both.
For each new segmentation algorithm, we will test performance on both real and synthetic data,
measuring
 accuracy of segmentation, by comparing voxel labels against ground truth and against competing
algorithms;
 range of initial conditions for which algorithms converge to a common answer;
 robustness of segmentation to the presence of noise in the input.
For each new registration algorithm, we will test performance on both real and synthetic data,
measuring:
 accuracy of registration, by comparing to ground truth. This will include measures of deviation of
boundaries of the data, deviation of landmarks within the data, and maximal deviation of tissue
surfaces;
 capture radius of registration methods – that is, the range of variation in initial position of the data
sets for which the algorithm correctly converges;
 robustness of registration methods to data occlusion, and data noise;
 effects of local minima in the search process for registration.
We will also use perturbation studies to explore valid parameter ranges for algorithms. For example,
Ettinger used perturbations of real data sets to evaluate registration algorithms, determining the range of
variation in initial starting position tolerated by the algorithm, range of occlusion of data sets tolerated
by the algorithm, amount of corrupting noise tolerated by the algorithm, effects of variations in search
strategy on the algorithm's performance, effects of variations in optimization function on the algorithm's
performance. We will use the same framework to evaluate our new registration algorithms, and our new
segmentation algorithms.
Page 93
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
D.2.2 Algorithms
a) Segmentation
(1) Integration of Strategies for Adaptive Filtering and Classification
This is CF’s work. He should fill in
(a) Plan
Oversampling, multiple features, drive cost functions for classification / registration
What will we do in year 1
Year 2
Year 3
Year 4
(b) Testbed and Validation
What phantoms will CF use?
Which of the clinical testbeds will he use?
Knee: bone enhancement, vessels
Postate: noise reduction, contrast enhancement, vessels
Liver: thermal imaging
(2) Intensity Based Classification
This is for Sandy to work on
(a) Plan
Implement and validate Mean field approximation of Markov Random Field:
Year 2 and 3
(b) Testbed and Validation
What phantoms, which standards? Base on the validation work in tina Kapurs thesis
(3) Segmentation by Alignment
(a) Plan
Our research has previously investigated several different segmentation strategies separately and in
the context of clinical applications in the brain. The focus of our previous segmentation work has been
the design of adaptive classification algorithms, incorporating the modeling of random noise, intensity
artifacts and normal anatomy. The primary clinical applications have been in quantitative brain analysis,
with some orthopedic work. We propose to integrate the range of new segmentation methods into this
framework, then apply and validate these methods in the context of segmentation of the prostate and the
liver.
The novel research presented here involves the integration of these strategies in a single framework
and their extension and validation for non-brain applications. We will design, implement and validate
algorithms which apply spatial constraints to the tasks of: rejection of random noise, removal of
smoothly varying intensity artifacts, enhancement and localization of structure boundaries, and the
spatial adaptation of likelihood and prior probability terms to exploit our ability to model anatomy. The
goal is to develop a complementary suite of algorithms that create patient-specific 3D anatomical
models from medical imagery. The specific objectives of this part of the proposal are to:
Page 94
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
1. Develop, implement and validate a Mean Field (MF) approximation to Markov Random Field
(MRF) regularization of random noise in an Expectation-Maximization (EM) framework. By
extending our EM framework for statistical classification in this manner, we will remove noise
and smoothly varying intensity artifacts and exploit voxel neighborhood statistics, thereby
improving the accuracy of the segmentation;
2. Develop spatially varying prior (SVP) models by empirical observation of the distribution of
structures across a range of subjects, and then implement and validate maximum a posteriori
(MAP) classification using these spatially varying prior models in the areas of prostate and liver.
This will allow prior anatomical knowledge to guide local decisions on tissue labels, extending
the accuracy of earlier methods.
3. Develop, implement and validate posterior probability smoothing based upon length-, area- and
volume-based curve evolutions (CE). This will complement segmentation based on statistical
classification, and will incorporate global knowledge of structures into the local labeling of
voxels.
4. Develop, implement and validate template moderated classification (TMC) based on nonlinear
registration of a template of normal anatomy to each subject. This approach uses anatomical
knowledge of normal structures to automatically provide local context for voxel classification.
5. Develop, implement and validate a classification approach integrating MF-MRF, SVP, CE and
TMC in an EM framework. This will leverage a range of segmentation tools in a coherent way.
Accomplishment of these objectives will significantly improve the sensitivity and specificity of
automated segmentation and will expand the range of medical image analysis problems that can be
considered candidates for automated segmentation.
(i) Motivation for Plan
In this section our previous approaches to classification are summarized to place them in context with
one another and to show their complementary features. It is shown that each of these algorithms, which
have been developed separately, can be regarded as adapting different terms of a traditional
classification algorithm. This allows us to formulate a clear strategy for generalizing our segmentation
framework to cover a broad range of applications.
The usual approach to identify the type of tissue present at a given voxel, is to assign a label for the
tissue based upon the most likely class, given the observed intensity. Let the observed intensity of a
voxel at spatial location i be v. Let Pr(Vi = v|Ci = c), model the distribution of observed voxel intensities
where i is an index ranging over all voxels in the data set, Vi is the value of the pixel and Ci is its class.
Given a set of intensity distributions Pr(Vi = v| Ci = c) and priors Pr(Ci = c) (i.e. probabilities of different
tissue classes occuring in the imagery), Bayes' Rule from elementary probability theory can be applied
to calculate the posterior probability that a given pixel belongs to a particular class, given its intensity.
Our early approaches to segmentation used precisely this approach. The intensity distributions Pr(Vi
= v| Ci = c) were modeled as sums of Gaussians. The EM algorithm iterates between a step (the E-step)
involving estimation of the posterior probability of voxel i belonging to tissue class c in the presence of
an assumed intensity artifact, and estimation of a smooth intensity inhomogeneity artifact given an
assumed tissue classification (the M-step). [CITE EM PAPER] Connectivity information and manual
editing were used to correct for errors in the segmentations generated with this approach.
Our MRF work for improved noise rejection has involved investigating adaptive approaches to
estimating the prior Pr(Ci = c). In the original EM formulation, prior probabilities were stationary, but
this is not always sufficient. An MRF approach to let local neighborhoods influence the prior
Page 95
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
probability of tissue classes being present based upon observed neighborhood statistics [CITE KAPUR
MRFs, HELD paper]
Our template moderated classification work further develops the use of geometric information, by
introducing the concept of using an anatomical template to modify the feature space in which the
classification is carried out. This method effectively allows spatial adaptation of the intensity
distributions Pr(Vi = v| Ci = c) to make the classification of voxel i more likely for the class which
appears in the anatomical template at location i and less likely for other classes. This allows
segmentation of structures with similar intensity properties [CITE WARFIELD MedIA PAPER].
The methods described above are all statistical in nature. We can augment these approaches with
knowledge of object shape to guide segmentation in uncertain regions. An initial method of combining
statistical and curvature driven methods for segmentation uses curvature evolution to smooth the
posterior probabilities [Haker99], [Teo98]. We can calculate the posteriors Pci = Pr(Ci = c|Vi = v) and
smooth by evolving Pc according to the affine geometric heat flow equation, under which the level sets
of Pc undergo affine curve shortening whilst preserving edges [Alvarez93], [Angenent98], [Sapiro94]. It
is very important to note that this approach has a very strong connection to Markov random field (MRF)
techniques. Indeed, the prior smoothing of the posterior probabilities gives the MAP solution to a
discrete MRF with a non-interacting, analog discontinuity field. The combination of a discontinuity field
with a discrete MRF can have some important consequences since it allows the disabling of clique
potentials across discontinuities. This is in contrast to the isotropic (linear) smoothing of the posterior
probabilities, which corresponds to computing the MAP solution of a single discrete MRF using
continuous relaxation labeling.
We also intend to explore area- and volume- based metrics for curvature driven evolution
[Siddiqi98]. Again, Bayesian statistics may also be built into the weighting factors, which are an integral
part of this method. Adaptive strategies of the type referred to above may also be applied to learn the
proper metric for the application at hand. Thus we see that the statistical, Markov random field,
anisotropic diffusion, and curvature driven flow approaches to segmentation are intimately related.
Further, as we have already discussed, a number of the ideas we have suggested have roots in the
pattern-theoretic framework of Grenander and his colleagues [Grenander1976]. The ongoing work on
the methodology of deformable models (see e.g., [Amit91], [Miller93], [Grenander1998] and the
references therein) will continue to influence our research and development of the most efficient
algorithms for the segmentation of medical imagery.
(ii) Illustration of Clinical Application of the Plan: Prostate
Brachytherapy
The space available does not allow detailed explanation of each of the clinical applications we will
investigate. This explanation of our proposed approach to prostate segmentation is illustrative of the
approach that we will apply to other specific clinical problems.
MRI scans of the region of the prostate are obtained with an endorectal coil in an open-magnet
scanner. The primary acquisition types lead to T1 weighted and T2 weighted images. These images
provide complementary contrasts and together provide sufficient contrast for the recognition of different
structures and brachytherapy seeds. The FOV encompasses muscles, the prostate (consisting of a central
zone and a peripheral zone), the urethra and the rectum. The images have a smoothly varying intensity
artifact due to the nature of imaging with an endorectal coil. The artifact is apparent as an increase in the
signal intensity nearby the coil and a smooth decrease across the FOV to dark signal at the periphery of
the FOV. The images are also affected by the usual random noise fluctuations present in conventional
MRI.
Page 96
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
The current clinical application of image acquisition is surgical planning and surgical guidance.
Surgical planning occurs intraoperatively, with manual outlining of the peripheral zone, central zone,
urethra, part of the rectum and the tumor. The use of manual outlining during surgery is a significant
burden. It would by far be preferable to obtain an automated segmentation of these structures.
We will apply and assess the effectiveness of the above image segmentation methods in this context.
First, as part of an exploratory process, we will model the intensity distributions of the major tissues
through the selection of prototype voxels and generate tissue classifications. We will investigate the
ability to enhance contrast and improve segmentation by enhancement of the signal intensities with
adaptive filtering. We will then incorporate algorithmic refinements in order to improve the
segmentation and increase our ability to automate the segmentation. We expect that the intensity
inhomogeneity artifact will be captured through the use of the EM algorithm, and that MRF
regularization will provide random noise rejection.
We will record the segmentations obtained during surgery. After completion of twenty surgeries we
will use nonlinear registration to align the segmentations and we will construct an atlas of the spatial
distribution of tissue classes probabilities. This will be stored in a volume with each voxel consisting of
a vector whose elements count the number of times each tissue class has appeared at the spatial location
of the voxel. After the completion of each subsequent surgery we will incorporate the new segmentation
into the atlas. The use of this SVP atlas to increase segmentation accuracy by modifying the prior
probability of tissue classes during surgery will be investigated.
It would be convenient if a segmentation derived from a preoperative scan could be used instead of
requiring an new segmentation intraoperatively. We will investigate the use of nonlinear registration to
project a preoperative segmentation onto intraoperative images. We will investigate the accuracy
obtained by using template moderated classification (TMC), with the template derived from the
preoperative segmentation, both alone and in conjunction with EM-MRF-SVP adaptation.
We will investigate the segmentation improvements obtained by applying curve evolution based
smoothing to the posterior probabilities, both alone and in conjunction with EM-MRF-SVP-TMC
adaptation. We will also investigate the effectiveness of curve evolution for localizing the peripheral and
central zone boundaries.
(b) Validation of Developments During the Execution of the
Plan
The validation of algorithms for medical image segmentation is made difficult by the fact that the
true segmentation of medical images is usually unknown. The construction of phantoms or synthetic
images for which the true segmentation is known must be done carefully to ensure the model reflects
human anatomy and imaging characteristics lest the segmentation problem becomes too simple. We
propose to measure accuracy and reproducibility with a set of tests which will be applied to each of the
components and to the entire system. We will test and validate the usefulness of each component.
The validation approach will consist of:
 The algorithms will be validated by comparing the results to ``brute force segmentation'' by a
human operator. The time required for performing the segmentation for each of the components
will be documented and the results, both intermediate and final, will be stored in a dedicated hard
disk. The data from approximately 20 patients will be processed in this way. This database will
then serve for validation purposes, when the improved algorithms become available.
 Validation of the algorithms operation will be evaluated on synthetic data sets specifically
designed to determine the limits of the algorithms applicability. The effect of varying levels of
white noise and intensity inhomogeneity artifact will be evaluated. This will be done by
Page 97
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
computationally adding known levels of white noise and known intensity inhomogeneity profiles
to validation images (derived synthetically and through phantom studies), and then segmenting
these images.
 The evaluation of the quality of the resulting segmentation will be done by comparing to ground
truth derived from rater segmentations, phantoms and synthetically constructed images.
Algorithm quality will be assessed by measuring accuracy, sensitivity and specificity of the
segmentation, as well as by measuring the (intra- and inter- rater) reproducibility of the
segmentation method.
We will use different validation approaches in order to ensure a comprehensive evaluation of the
algorithms we develop. We will construct and evaluate segmentation on synthetic images with varying
levels of noise and intensity inhomogeneity artifact in order to assess robustness with respect to
instrument artifact.
The primary measure for the comparison of segmentations will be to count the number of voxels
where both segmentations have the same tissue class value and to count the number of voxels where
they have different tissue class values. This is a measure of spatial overlap.
Reproducibility of segmentation will be assessed for a variety of medical image analysis problems
with the following strategy. A particular scan will be segmented repeatedly and the coefficient of
variation of tissue class volume for each tissue class will be reported. Multiple scans of a particular
subject have been acquired in one session to simulate a longitudinal study. Reproducibility will be
assessed by segmenting each of these scans, allowing a direct measure of end-to-end reproducibility
including imaging artifact effects to be reported.
The comparison of new segmentation methods to existing segmentation methods, including manual
segmentation, on large numbers of scans will be carried out with the following strategy. Databases of
scans from a variety of medical image analysis problems will be segmented with the methods to be
compared. For each segmentation, volumes of each structure of interest will be recorded. A t-test will be
used to determine if a difference exists between the mean tissue volumes determined with each
segmentation method. The time required for operator interaction with each method will be recorded. The
compute time for each method will be recorded.
(c) Timetable:
Validation is an ongoing process that is integrated throughout the grant. As each component is
completed it will be validated using the approach described above.
Through years 3-5 construction of empirical atlas of SVP for prostate and liver as segmentations
become available through the clinical routine.
 Year 2: MF-MRF prior implementation in 3D and integration into EM.
 Year 3: integration and validation with CE smoothing of posterior probabilities.
 Year 4: initial evaluation comparing TMC and EM-MRF using existing nonlinear registration for
TMC.
 Year 5: integration and validation of TMC-EM-MRF approach, evaluation of new non-rigid
registration methods for TMC.
(4) PDE’s and Snakes
We have experimented with an initial method of combining statistical and curvature driven methods
for segmentation using curvature evolution to smooth the posterior probabilities [Haker99], [Teo98]. We
can calculate the posteriors Pci = Pr(Ci = c|Vi = v) and smooth by evolving Pc according to the affine
Page 98
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
geometric heat flow equation, under which the level sets of Pc undergo affine curve shortening whilst
preserving edges [Alvarez93], [Angenent98], [Sapiro94]. It is very important to note that this approach
has a very strong connection to Markov random field (MRF) techniques. Indeed, the prior smoothing of
the posterior probabilities gives the MAP solution to a discrete MRF with an non-interacting, analog
discontinuity field. The combination of a discontinuity field with a discrete MRF can have some
important consequences since it allows the disabling of clique potentials across discontinuities. This is in
contrast to the isotropic (linear) smoothing of the posterior probabilities which corresponds to
computing the MAP solution of a single discrete MRF using continuous relaxation labeling.
We also intend to explore area- and volume- based metrics for curvature driven evolution
[Siddiqi98]. Again, Bayesian statistics may also be built into the weighting factors which are an integral
part of this method. Adaptive strategies of the type referred to above, may also be applied to learn the
“proper” metric for the application at hand. Thus we see that the statistical, Markov random field,
anisotropic diffusion, and curvature driven flow approaches to segmentation are intimately related.
Further, as we have already discussed, a number of the ideas we have suggested have roots in the
pattern-theoretic framework of Grenander and his colleagues [Grenander1976]. The ongoing work on
the methodology of deformable models (see e.g., [Amit91], [Miller93], [Grenander1998] and the
references therein) will continue to influence our research and development of the most efficient
algorithms for the segmentation of medical imagery.
(a) Plan
(i) Year 1: Adaptive Learning, Noise Models, and
Geometric Active Contours
There are a number of directions which we will consider in the knowledge-based segmentation work
which we just outlined above. So far, we have only considered simple prior distributions and adaptation
techniques. For difficult noisy data, it is possible to introduce more sophisticated mutliscale texture
models. Another extension would be to consider that the number of classes in the image is not given and
needs to be estimated as well. This could be accomplished via EM type algorithms. In [Haker99], we
assumed that the pixel intensities are distributed according to fixed normal distributions. Instead, we
plan to learn the sample distribution of densities within each class as images are segmented.
We also will incorporate the Bayesian type statistics into the stopping (conformal weighting ) factor
in the geometric active contour model [Kichenassamy96]. At this point the weighting factor is only
local based on edge computations. A more accurate conformal metric will be obtained if the metric is
learned from the data and if it models non-local information. One can also tailor the weighting factor to
segment such vector-valued features such as textures or color. Textural segmentation (with conformal
metric based on Fourier or wavelet analysis) may be quite useful for noisy medical image data. Finally
the snake model in the latter reference is based on the minimization of length in the plane and surface
area in space. We will also test the other flows based on the miminization of other natural geometric
quantities (such as area in the plane) as described in [Siddiqi98]. Here once again, we will investigate
other types of weighting factors for vector-valued features.
(ii)
Year 2: Active Regions
There has been some interesting work on attempting to explicitly couple boundary and region
data in the geometric active contour framework; see [Paragios99], [Yezzi99], and [Zhu96]. Moreover,
this work lends itself quite naturally to incorporating statistical models of the Bayesian (or minimum
Page 99
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
description length type) into a PDE framework. In particular, the technique in [Zhu96] is based on the
concept of minimum description length (MDL). The idea is to consider the segmentation problem as a
partitioning problem, where the criterion for choosing one partition instead of another is the description
length. As is usual, the measure of the description length must be accomplished according to some a
priori language. Thus in fact, it is essentially equivalent to the maximum a posteriori (MAP) estimate
from the Bayesian paradigm. It may be regarded as an information interpretation of this classical
method. In [Zhu96], the authors propose an MDL based energy functional. Taking the gradient
direction, one obtains a system with a smoothing term based on Euclidean curve shortening, and a term
which determines the motion of a point on the common boundary of two regions via the likelihood ratio
test. (In this sense, the model of [Zhu96] is closely related to the “bimodal image” work of [Yezzi99].)
The “geodesic active region” framework of [Paragios99] treats the supervised texture
segmentation problem. Boundary information is determined using a probabilistic framework, while
region-based information is treated using conditional probabilities. A very important feature of this
work, is that it may be implemented using level set techniques, which gives it the ability to
automatically change topology. These methods may be quite useful in segmenting such data as the
prostate and even gray/white brain boundary.
All of the above related methods fit very naturally into our overall research paradigm of
combining curve evolution with statistical techniques. They will be carefully implemented and tested in
order to develop the optimal combination of the statistical and PDE-based segmentation approaches.
(iii)Year 3: Addition of Temporal Information into
Segmentation
Many images are dynamic, that is, they change in time. Therefore an important segmentation
project would be to take temporal information directly into account in the segmentation process. The
Bayesian/ anisotropic diffusion approach in [Haker99] is a possible starting point is this direction.
More precisely, when segmenting sequences of images, we can extend the model so that information
from one frame is used in the segmentation of the next one. By far the most effective way we have
found to do this is by modifying our initial assumption of homogeneous priors. In particular, we can
learn these priors, by employing the smoothed posteriors from one frame as priors in the segmentation
of the next.
One can also try to incorporate explicit temporal data (e.g., optical flow) in the energy functional
defining the gradient descent PDE in the geometric active snake context. The curve evolution approach
to optical flow as given in [Kumar96] would fit very naturally into this methodology. In fact, both of the
functionals in [Kichenassamy96] and [Kumar96] are based on modifications of Euclidean curve
shortening.
The active region ideas in [Paragios99a] may also be very relevant. Here the detection and the
tracking problem is treated using geodesic active contours with the incorporation of region-based
measurements. Region-based features are estimated from a certain motion detection/segmentation
module. Boundary and region information are coupled leading to the geometric active region model.
Further the authors add a visual consistency module which is based on an affine motion model. The
flows obtained here fit in quite well with the statistical/curve evolution approaches described above, and
can be efficiently implemented using level sets. Thus we will be exploring the use of such
boundary/region ideas for the incorporation of temporal data into our proposed segmentation work.
Page 100
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
.
Expand Lianas stuf to work on vessels in the body
Use allen/andi tsai stuff on prostate, liver, knee
Integrate the gibson/leventon style active surfaces
(b) Testbed and Validation
b) Non-Rigid Registration
(1) MI based Non-Rigid Registration
This is for Sandy to work on. The time table called it general purpose non-rigid registration. Since
there is also a registration for template driven segmentation under that heading, I assume that this is the
sandy corner while c is the simon corner. Guys, please talk to each other.
(a) Plan
This is supposed to be work during Y1-3
(b) Testbed and Validation
What phantoms?
Which testbeds?
(2) Seed Based Registration for Brachytherapy
Noby will write this section
(a) Plan
Work on it for year 2-4
(b) Testbed and Validation
What phantoms, which testbed?
(3) Shape Modeling, Representation, and Deformation
(a) Plan
In the research plan we have outlined below, we describe several strongly interrelated projects. Some
of the research on the projects will occur concurrently, but we will emphasize the main focus in each of
the initial periods of support.
(i) Year 1: Area-Preserving Maps of Minimal Distortion
We have already mentioned our work on brain flattening for fMRI in [Angenent99a]. Using the
Laplace-Beltrami operator, one constructs a mapping from the brain surface to the sphere which is
bijective and conformal (angle-preserving). The procedure is implemented via finite elements on a
triangulated surface. In [Angenent99b], we have considered a completely new class of maps which are
area-preserving of minimal distortion, that is in the class of area-preserving mappings they optimally
preserve angles. (Unless the two surfaces are diffeomorphic and have the same Gaussian curvature, there
is no simultaneous angle- and area-preserving bijection.) The deformation is again based on a certain
partial differential equation which can be implemented using finite elements. We plan to explicitly
implement in first year of the proposed research program.
For both the conformal and area-preserving of minimal distortion approaches, one can derive
bijective mappings from one diffeomorphic surface to another, simply by mapping through a canonical
intermediary of the same topology. In the context then of brain imagery , this will allow us to map one
brain surface onto another, and one brain structure onto the corresponding one of another brain. Clearly,
Page 101
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
this observation holds for any given anatomical structure. This presents then a very natural metric of
distortion which could be applied to study any number of pathologies. Registration then becomes a very
natural corollary of this approach.
We indicate a conformal mapping of the brain surface (with a tumor) to a sphere, as well as the
gray/white matter boundary to a sphere in Figure 1ALLEN. The gray/white matter segmentation was
performed using [Teo98] and [Haker99]. The points are colored according to mean curvature with the
lighter points having the higher mean curvature.
(C-F: I have included four images: top_b.tif, top_s.tif, white_left_brain.tif, and
white_left_flat.tif. The two files white* consist of the white/gray matter boundary
and corresponding flattened surface, and the two files top* consist of the cortical
surface and the corresponding flattened surface. Ron asked for some nice images.)
(ii) Year 2: Noise-Resistant Skeletons
We have already alluded above several times to the use of skeletons as a powerful shape descriptor.
The problem once again is that noise in real images makes it necessary to use various types of “fixes”
(many of them quite ad hoc) in attempting to use the skeleton as a reliable and robust description of
shape. The classical skeleton is based on the usual Euclidean distance (in the plane or in space). In some
very new work, we have proposed replacing the classical Euclidean distance by an affine invariant one
based on areas in 2D and volumes in 3D [Betelu99]. As evidenced by extensive simulations on real and
synthetic imagery, this skeleton seems quite robust to noise, and to small deformations in the shape of
the object. The reason for this is that the area or volume-based distance of affine differential geometry
averages out small variations in the boundary of the given object being skeletonized. The other key
advantage of such a skeletonization procedure is that it is affine invariant. This means that if a
unimodular (area-preserving) linear transformation is applied to the boundary of the shape, the affine
skeleton transforms precisely according to the same transformation. Thus it is invariant to various types
of shears, as well as rigid linear transformations as in the classical case. These properties make the affine
skeleton a natural candidate to be employed in statistical shape analysis for the study of brain pathology
as in [Golland99]. Since the affine skeleton is much more robust to small noise variations than the
classical skeleton, this procedure could be used to detect larger shape deformations which are actually
connected to salient features, and are not artifacts of noise or numerics.
(iii)Year 3: Dynamic Shape Analysis
One can consider another kind of deformation of the skeleton and the original shape which we plan to
test in shape analysis for several anatomical structures in medical imaging. Namely, one can pose the
problem of determining how the skeleton changes for an object changing (evolving) in time. Indeed, this
question clearly has major practical implications since given the instability of the classical skeleton to
boundary perturbations, smoothing of the boundary is typically employed in the actual computation of
the skeleton. Clearly, rapid object recognition seems to require hierarchical shape representations to
organize a database, and the instability of the skeleton supports the view that such hierarchies cannot be
reliably computed. In [Bruce86], there is a very general discussion of how the skeleton changes under
generic perturbations of the original shape.
In some very recent work [August99], we have given an explicit description of how the skeleton
evolves when the original shape is smoothed using the geometric heat equation. Interestingly, even
Page 102
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
though the original curve is being smoothed, new branches may appear in the skeleton, that is, the
skeleton may not be getting monotonically simpler. Therefore in addition to the mappings above for
shape, we also propose to study the skeleton under various classes of deformations in relation to key
structures for the proposed medical imaging applications to the brain, prostrate, and the liver. Again
statistical analyses of the type described in [Golland99] and [Martin98] are very relevant. Moreover, one
can consider the affine invariant skeleton just described, and affine invariant perturbations for it, e.g.,
evolving it with the affine invariant heat equation [Angenent98], and studying its changes. A key new
shape descriptor which could come out of this study is the precise time in which the number of branches
of the skeleton start to decrease monotonically. A careful treatment of this task will impact work in
dynamic shape analysis for various anatomical features and feature extraction.
(iv) Year 4: Invariant Shape Recognition
Invariant shape recognition is a central task in computer vision. In some preliminary work
[Calabi98], we have proposed a new paradigm for the group invariant recognition of visual objects
based on the theory of differential invariants. This work relies on the concept of group invariant
manifold and a new methodology we have found for the robust computation of differential invariants;
see [Calabi96] and [Calabi98]. In conjunction with some of the geometric active contour methods
described above as well as a curve evolution approach to optical flow [Kumar96], this paradigm also
may be used for dynamic object tracking. In general, shape invariants are essential in object recognition
since they are independent of the viewpoint in which the given object is observed. An important feature
of visual recognition is the ability of the detector to find symmetries of objects or between two similar
objects. The skeleton to which we have referred several times previously is precisely used to find such
symmetries. Many times recognition requires the identification of distinguishing landmarks. This has
been alluded to for the problem of registration. In the context of signature manifolds, recognition under
group transformations would reduce to the determination of whether two computable associated
manifolds (or portions thereof) are identical or not. It is important to emphasize that these manifolds
have a rather special form. Symmetries of an object are then found in terms of a winding number
associated to the corresponding signature manifold.
(v) Year 5: Integrated System for Shape
We will integrate the above ideas into a working system for analyzing anatomical shape during the
final year of the proposed research program. In particular, we will show how to formulate a completely
invariant object extraction, modeling, and shape recognition system. This will even include invariant
segmentation for feature extraction. This in conjunction with the deformation and statistical approaches
already discussed will be a key component of our research. Even though the outline we give is valid for
most of the key symmetry groups of interest in vision we will concentrate on the affine group where
much of the available research has been done so far. (Euclidean invariance is usually taken as a given in
image processing.)
The first part of such a program will be the derivation of geometric feature detection and image
denoising schemes which incorporate invariance. By this we mean that if two images are related by an
affine and photometric transformation, the images and object boundaries or edge maps obtained by the
algorithms are also related by the same transformation. This was a key property of the affine skeleton
which we just described. Affine smoothing can be accomplished by the use of the invariant smoothing
filter as in [Alvarez93], [Sapiro94], and [Angenent98]. This would be followed by the computation of
affine invariant descriptors. This work has very important practical interest if one wants to use even
simple invariants in recognition. Indeed, incorporating such invariant detection and denoising schemes
Page 103
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
in object recognition systems will reduce the ``algorithmic'' noise introduced by using non-invariant
methods. Smoothing an image with a non-invariant algorithm before computing the invariant curvatures,
will introduce ``artificial noise'' into the computation, and when comparing the signatures of different
shapes, it would be very difficult to know if the differences correspond to the shapes or were introduced
by the algorithm.
The next step in a shape recognition system is to recognize the objects bounded by the edges that
have been detected by a segmentation procedure. For this purpose, as we have discussed above, new
approach incorporating general invariance, based on the general concept of a differential invariant
signature manifold, was recently proposed in [Calabi96] and [Calabi98]. These papers derive novel fully
invariant numerical algorithms for computing the invariants relative to general symmetry tranformation
groups required to uniquely characterize the feature up to a given group tranformation, and hence, in
conjunction with the aforementioned algorithms, form the basis of a fully group-invariant object
detection and recognition procedure.
We have proposed in [Olver99] two different affine edge detectors. The first one is derived by
weighted differences of images obtained as solutions of the affine invariant scale-space developed in
[Sapiro93]. The second one is given by the simplest affine invariant function which shows behavior
similar to the magnitude of the Euclidean gradient. These affine invariant edge maps can then used in to
define invariant snakes extending the work in [Kichenasamy96] and [Caselles97]. For some preliminary
results along these line, see [Olver99]. The active contours will be used to integrate the local information
obtained by the invariant edge detectors. In this case, the boundaries of the scene objects lie at the
bottom of a potential well relative to a geometrically defined energy functional based upon the affine
geometry of the plane weighted by an affine invariant stopping function. Noise resistant skeletons based
on the affine distance fit naturally into this framework as a key invariant and robust descriptors of
shape.
Thus we will fashion the algorithms we have described above into one fully integrated system for the
shape modeling, representation, and deformation of key anatomical features in this research.
(b) Testbed and Validation
We plan to test the algorithms above on several different testbeds including brain, colon, liver, and
prostrate . In principle, they can be applied to any given anatomical structure. We have already
discussed MR imaging of the brain in some detail. We plan to use the flattening procedure in functional
MR studies of the brain since we can now unfold the cortical surface as well as the gray/white boundary
in order to detect neural activity in the sulci. Moreover, we plan to employ the conformal and areapreserving procedures to map and compare brain surfaces, as well as mapping the brain to a canonical
atlas in order to develop shape measures to detect possible pathology. Notice that we can map any type
of geometric information from one structure to the other including curvatures, depth information,
normals, etc. In the conformal case, we can even compute how intrinsic distances change from one
surface to the next, which is essential is surface mapping and registration. The skeletonization methods
apply for feature extraction, and shape matching. They can be combined with statistical considerations
as well.
We will also consider applying the flattening technique to the CT and MR images of the colon for
virtual colonoscopy. Such visualization methods have the potential for non-invasively determining the
Page 104
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
presence of polyps and other topologies. Further we will test our method of providing a surface scan of
the entire colon as a cine, and then provide the viewer the opportunity to examine each point on the
surface without distortion [Haker99a]. Thus we will provide a colon “fly-through” with 100% view and
0% distortion. If we color the points on the colon surface according to Gaussian curvature, then polyps
should become visible since they are roughly of spherical shape of high Gaussian curvature while the
usual surface of the colon should have much smaller curvature.
The liver is a common site for cancer metastasis. Many cancers can spread to this organ such as
breast cancer, colon cancer, stomach cancer, ovarian cancer, kidney cancer, malignant melanoma,
esophageal cancer, testis cancer, and choriocarcinoma. Symptoms associated with this condition vary,
depending on the extent of liver involvement. Most patients are diagnosed with liver metastasis on
routine work-up of the cancer, when there are no symptoms at all. An MR, CT , or ultrasound scan can
many times establish the diagnosis. It would be natural to register these modalities using the techniques
we have just described. Statistical based curvature analysis could be used in comparing normal and
abnormal organs. Again for the prostate, these methods could be used in shape analysis and registration.
D.2.3 Software Engineering
The entire section called Software Engineering should be written by Mike Halle.
a) Distributed Version Control for Software Modules
(1) Plan
Year 1-4
b) VTK modules for Communication with Networks and Data
Sources
(1) Plan
Year 1-2
c) Incorporate DICOM Reader into the Visualization Pipeline
(1) Plan
Year 2-3
d) Develop XML Based Data Structure for IGT
(1) Plan
Year 2-4
e) Enhancements to the Visualization Pipeline
(1) Plan
Volume rendering
Surface subdivisions
Year 3-4
D.2.4 Application Validation and Quality Control
The ultimate test of IGT technology is in the development of complete systems that can be applied to
specific surgical procedures. Building on our earlier experience in creating image guided neurosurgical
systems (both in conventional ORs and in interventional or intraoperative MRI scanners), we will
develop a series of IGT systems, including systems for MRI-guided laser thermal ablations and prostate
brachytherapy. These projects are actually under clinical investigation, however, there is now a
Page 105
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
relatively low level of integration. We will continue to merge our image processing, surgical planning
and visualization tools and incorporate them into the therapy process. This eventually will lead us to the
development and implementation of complete, integrated therapy delivery systems, which can function
more effectively then the current relatively disjointed system.
Validation of our advances in components and systems for Image Guided Therapy will
require careful exploration on four different fronts: evaluation of individual algorithms,
evaluation of integrated systems, validation of software engineering, and surgical performance
evaluation.
a) System Evaluation
A systematic evaluation of the reproducibility and consistency of the entire procedure and of
its individual components is also necessary. In earlier work, we assessed the accuracy of
volumetric measurements of MS lesions [Guttmann98]. In that disease MRI is impeded by the
difficulty of having a reference measurement of the "ground-truth" from independent
measurements, such as the identification of lesions on corresponding histological sections of
the imaged brains. A similar situation can arise in the case of tumors. Therefore, at the present
stage, the major requirement for quantitative assessment of lesion size or lesion load is the
measurement reproducibility. No phantom is able to reproduce satisfactorily the complexity of
the brain and the lesions, their relaxation characteristics, shape and spatial distribution. We
therefore elected to perform MRI repeatedly on each of 20 MS patients within very short time
intervals, during which lesion volumes were not expected to change. Statistical analysis of
measurements obtained by applying various combinations of the image processing chain was
performed to assess strengths and limitations of this approach to lesion burden estimation.
Similar evaluation of the image-processing pipeline will be obtained during this proposed
research, and the approach will be extended to other application areas.
b) Performance Evaluation
In order to provide measurements to objectively and subjectively document and guide the
development of our image guided-program, we need to measure the impact of the system. To
do this, we will consider the amount of time spent on the computer and the imaging in relation
to the surgical time, as well as the subjective evaluation of the system by computer operator,
surgeon and neuroradiologist.
(1) Computer Measurements
We need to determine the role and impact of the computational tools in delivery IGT therapy. One
aspect of this is the actual computational time expended in relation to more traditional surgical methods.
We can add software instrumentation to our system to record such timing information.
The surgeon or the operator at the console may run Slicer. Switching and the time spent in that
specific mode are recorded automatically. Idle time is recorded as well. Since each function/tool is
invoked from the program (Tcl/Tk) a time stamp, using the system time, allows the exact reproduction
of the time spent on each feature. This allows us to measure the following factors:
1. Preparation: The set-up of the computer and the software as well as the necessary preprocessing of
the data will be included in this section. Parts of the processing of the data are addressed in a
different grant.
2. Operator driven: The frequency and the time that the system is used by the operator, either to set-up
the display for specific surgical questions or for processing the newly acquired data.
Page 106
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
3. Surgeon driven: The frequency and the time that the system is actually driven and used by the
surgeon.
(2) Surgical Time vs. Scanning Measurements
As a second objective measurement we will record the surgical time expended. Dural opening and
closure frame the time during which localization and image-guidance is crucial. Thus we concentrate on
the period with our time measurements. During this period we will record the time spent on imaging,
and application of the Slicer. The objective is to analyze if there is a marked reduction of scanning time
and thus of overall surgical time due to computer-assisted intra-operative MR guidance
In order to achieve this goal we will apply the statistical methods detailed in Section on statistical
analysis.
(3) Questionnaire
Feedback from the users will be essential in guiding refinements to the IGT systems. We will
use a set of questionnaires from each procedure to determine impact and utility of our tools.
(a) Computer operator
The ease of use of the graphical interface will be targeted in this set of questions:
1. Did the software allow swift answers to the surgical questions?
Yes or No
2. Was the software fast enough?
Yes or No
3. How many errors/ crashes took place?
0; 1-5; 5-10; > 10
4. Was the software easy to use?
Yes or No
5. Is the interface feasible?
Yes or No
6. What features were demanded the most?
Narrative
The close communication of operator and programmer will allow a constant evolution of the
interface. Nevertheless the major developmental steps allow the division into different stages.
Thus the previous of these successive stages can be compared to the following stage,
functioning as a control.
(b) Surgeon
The subjective issues concerning the surgeon will be covered in the following way.
(i) Intra-operative questions
On requesting an intra-operative scan, the surgeon will be asked for the reason for this
scan, and for what he expects as an answer.
(ii) Post-operative questionnaire
A) Overall impression of surgical strategy:
Did you achieve your surgical goal? Yes or No
Did you have to change your approach?
Yes or No
B) Overall impression of the imaging:
How much did you rely on the imaging?
10 - 100 %
How comprehensive did you find the imaging?
Very/ good / not at all
Did the imaging supply new information?
Yes or No
Did the imaging augment the perception of the anatomy
Yes or No
C) Evaluation of the Slicer:
Page 107
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
How much did you rely on the Slicer? 10 - 100 %
How comprehensive was the Slicer? Very/ good / not at all
Did the Slicer supply new information?
Yes or No
Did the Slicer augment the perception of the anatomy
Yes or No
D) Comparison Slicer vs. MRI:
Which modality was easier for localization? MRI or Slicer
Which modality allowed for a better appreciation of the anatomy? MRI or Slicer
Which modality allowed more accurate appreciation of tumor resection?
MRI or Slicer
Which modality did you rely on in decision making? MRI or Slicer
E) Neuroradiologist:
What is the extent of Tumor removal? 10 – 100 %
Was the surgical goal achieved?
Yes or No
Was the strategy altered due to the MRI information? Yes or No
Was the strategy altered due to the Slicer information?
Yes or No
(4) Statistical Evaluation: Power Analysis of Sample Size for Time
measurements
In order to achieve a significant basis for the statistical tests mentioned in the following
section we performed a power analysis, which is based on 20 randomly selected cases.
Since the surgical time often has a skewed distribution, we focus on the time on a log scale. The
mean of the pilot data on a log scale is 5.6 with a standard deviation of 0.35. This log time corresponds
to 280 minutes or 4:40 hours. We assume that the control sample for our study has the same mean log
time, and the target data has the same standard deviation (0.35) as in the collected pilot sample. In
addition, let the ratio of the number of patients in the control vs. the number of patients in the target
groups be k=1:1 with a type I error fixed to be 5%. Further we assume that the mean log surgical time
under the target procedure is 5.50 (SD 0,35), which corresponds to 245 minutes or 4:15 hours (i.e., a
reduction of 35 minutes in surgical time). We then conduct the sample size calculation using a twosample t test for the difference between the mean log surgical times under the procedures guided solely
by MRI and the proposed target with the computer-assisted MR-procedures. If the ratio of samples of
the control and target groups is k=1:1, we need to enroll 100 patients in each group (control and target
group) in order to assure the power of 85% and type I error of 5%. In order to achieve the power of 90%
and type I error of 5% we need to enroll 120 patients in each group. With an annual rate of 75-100
patients/year, it requires 3-3.5 years to recruit the patients to complete this study.
The constant evolution of the computer interface and technology may allow a potential
improvement in the various measures of the computing abilities, for example, the speed of the software,
the feasibility of the interface and subsequently the surgical time. Since the systems and its integration
are in a developmental phase, the design of the study adopts a conservative approach. That is, we first
focus on the average improvement of the above-discussed measures. However we may also classify the
major development of our integrated system into various stages. We then will perform a statistical trend
analysis to examine how the various measurements depend on these different stages. Thus the
successive stage can also be compared with the proceeding stage, with the latter perceived as a control.
Page 108
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
(5) Analytic Strategy
The main goal of the statistical analysis is to compare the IMRI alone and in combination with
computer-assisted guidance (Slicer) in terms of optimizing surgical procedures in the intraoperative MRI
(IMRI).
Two objective measurements are performed: Time measurement on the computer; and Time
comparison for imaging and guidance during the operation. As a subjective evaluation for the overall
feasibility of the intra-operative MR image and the use of the Slicer is based on a questionnaire to
computer operator and surgeons.
To achieve these goals, the following statistical analysis methods will be employed:
1. Two-sample non-parametric and parametric methods (e.g., Wilcoxon rank-sum tests or two-sample
t-test of times after a possible appropriate transformation such as a log transformation of the data)
will be employed to conduct hypothesis tests to detect the differences between the various times
spent during MR guidance among the cases.
2. Multiple regression analysis of the "effective" time measurements versus a set of patient and lesion
characteristics will be performed to develop a statistical linear model to explore the differences and
the potential success in integrating the two guidance tools. Significant characteristics, such as size,
location and histopathology will be identified.
3. Summary statistics such as sample means, standard errors, and sample medians will be reported
containing information on the setup of the software, combinations and sequences used for
intraoperative MR guidance.
4. Analysis of counts and proportions will be used to report the detailed and overall results of the
questionnaire. The relationship between the success of achieving the surgical goal and the
appropriateness of using a Slicer is also assessed by non-parametric and parametric statistical tests
for independence (e.g., Fisher's exact test or Chi-square tests) between variables of interest.
5. Sequential analysis of the above outcomes. All measurements related to the intraoperative MR
guidance will be assessed periodically and sequentially stratified based on the different
developmental stages of the new technology.
D.3. Timetable
Y1
Y2
Y3
Y4
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Y5
Algorithms
Segmentation
a) Integration of Strategies for
Adaptive Filtering and Classification
b) Intensity Based Classification
c) Segmentation by Alignment
d) PDE’s and Snakes
Non-Rigid Registration
a) MI based Non-Rigid Registration
b) Seed-Based Registration for
Brachytheraphy
c) Registration for Template-Driven
X
X
Page 109
X
X
X
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
Segmentation
Software Engineering
1.) Distributed Version Control for
Software Modules
2.) VTK Network modules
3.) Dicom reader integration
4.) XML data structure
5.) Enhanced visualization pipeline
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Clinical Validation
1.) System Evaluation
2.) Performance Evaluation
3..) Software and Quality Assurance
X
X
X
X
X
Some more references for Tannenbaum’s write-up (these are new):
[Sapiro93] G. Sapiro and A. Tannenbaum, “On affine invariant scale-space,” International Journal of
Computer Vision, vol. 11 (1993), pages 25-44.
[Olver99] P. Olver, G. Sapiro, and A. Tannenbaum, “Affine invariant edge maps and active
contours,” to appear in Acta Math. Appl, 1999.
Additional References for Tannenbaum’s Write-Up (these were in the previous version)
[Betelu99] S. Betelu, G. Sapiro, A. Tannenbaum, and P. Giblin, “Noise-resistant skeletons of plane
curves,” Technical Report, Dept. of ECE, University of Minnesota, 1999. Submitted for publication to
IEEE PAMI.
[Blum73] H. Blum, “Biological shape and visual science,” J. Theor. Biology, vol. 38 (1973), pages
205-287.
[Bookstein89] F. Bookstein, “Principal warps: thin-plate splines and the decomposition of
deformations,” IEEE PAMI, vol. 11 (1989), pages 567-585.
[Bruce86] J. Bruce and B. Giblin, “Growth, motion and one-paramter families of symmetry sets,”
Proc. Royal Soc. Edinburgh, vol. 104A (1986), pages 179-204.
[Calabi96] E. Calabi, P. Olver, and A. Tannenbaum, “Affine geometry, curve flows, and invariant
numerical approximations,” Advances in Mathematics, vol. 124 (1996), pages 154-196.
[Calabi98] E. Calabi, P.Olver, C. Shakiban, and A. Tannenbaum, “Differential and numerically
invariant signature curves applied to object recognition,” International Journal of Computer Vision, vol.
26 (1998), pages 107-135.
[Caselles93] V. Caselles, F. Catte, T. Coll, and F. Bibos, “A geometric model for active contours in
image processing,” Numerische Mathematik, vol. 66 (1993), pages 1-31.
[Caselles97] V. Caselles, G. Sapiro, and A. Tannenbaum, “Geodesic snakes,” International Journal of
Computer Vision, vol. 22, pages 61-79.
Page 110
Principal Investigator/Program Director (Last, First, Middle): Jolesz, Ferenc A.
[Golland99] P. Golland, W. E. Grimson, and R. Kikinis, “Statistical shape analysis using fixed
topology skeletons: corpus callosum study,” IPMI’99 edited by A. Kuba et al., LNCS 161, SpringerVerlag, 1999, pages, 382-387.
[Kimia95] B. Kimia, A. Tannenbaum, and S. Zucker, “Shapes, shocks, and deformations, I: the
components of shape and the reaction-diffusion space,” International Journal of Computer Vision, vol.
15 (1995), pages 189-224.
[Koenderink90] J. Koenderink, Solid Shape, MIT Press, Cambridge, MA, 1990.
[Malladi95] R. Malladi, J. Sethian, and B. Vermuri, “Shape modeling with front propagation: a level
set approach,” IEEE PAMI, vol. 17 (1995), pages 158-175.
[Martin98] J. Martin, A. Pentland, S. Sclaroff, and R. Kikinis, “Characterization of neuropathological
shape deformations,” IEEE PAMI, vol. 20 (1998), pages 97-112.
[Ogniewicz93] R. Ogniewicz, Discrete Voroni Skeletons, Hartung-Gorre Verlag Konstanz, Zurich,
1993.
[Pentland91] A. Pentland and S. Sclaroff, “Closed-form solutions for physically based shape
modeling and recognition,” IEEE PAMI, vol. 13 (1991), pages 715-729.
[August99] J. August, A. Tannenbaum, and S. Zucker, “On the evolution of the skeleton,”
Proceedings of ICCV’99, Corfu, Greece, 1999, pages 315-322.
[Kumar96] A. Kumar, A. Tannenbaum, and G. Balas, “Optical flow: a curve evolution approach,”
IEEE Transactions on Image Processing, vol. 5 (1996), pages 598-611.
[Paragios99] N. Paragos and R. Deriche, “Geodesic active regions for supervised texture
segmentation,” Proceedings of ICCV’99, Corfu, Greece, 1999, pages 926-932.
[Paragios99a] N. Paragos and R. Deriche, “Geodesic active regions for motion estimation and
tracking,” Proceedings of ICCV’99, Corfu, Greece, 1999, pages 688-694.
[Zhu96] S. Zhu and A. Yuille, “Region competition: unifying snakes, region growing, and
Bayes/MDL for multiband image segmentation,” IEEE PAMI, vol. 18 (1996), pages 884-900.
[Yezzi99] A. Yezzi, A. Tsai, and A. Willsky, “A statistical approach to snakes for bimodal and
trimodal imagery,” Proceedings of ICCV’99, Corfu, Greece, 1999, pages 898-903.
[Wang98] G. Wang, E. McFarland, B. Brown, and M. Vannier, “GI tract unraveling with curved
cross sections,” IEEE Trans. Medical Imaging, vol. 17 (1998), pages 318-322.
[Haker99a] S. Haker, A. Angenent, A. Tannenbaum, and R. Kikinis, “Nondistorting flattening maps
and the 3D visualization of colon CT images,” Technical Report, Dept. of ECE, University of
Minnesota, 1999. Submitted for publication to IEEE Trans. Medical Imaging.
C-F: This last reference should replace the old [Haker99a] in the proposal.
Page 111