Download VTree-3D - UNC Computer Science

The Uses of Object Shape
from Images in Medicine
Stephen M. Pizer
Kenan Professor
Medical Image Display & Analysis Group
University of North Carolina
Credits: Many on MIDAG, especially
Daniel Fritsch, Guido Gerig, Edward Chaney, Elizabeth Bullitt,
Stephen Aylward, George Stetten, Gregg Tracton, Tom Fletcher,
Andrew Thall, Paul Yushkevich, Nikki Levine, Greg Clary,
David Chen
MIDAG@UNC
Object Representation in
Medical Image Analysis
 Extract
an object from image(s)
[segmentation]
 Radiotherapy
Tumor; plan to hit it
 Radiosensitive normal anatomy;
plan to miss it

PD
MRA
T2
T1
Contrast
 Surgery
Plan to remove it
 Plan to miss it
 During surgery, view where it is
& effect of treatment

 Radiology

View it to judge its pathology
MIDAG@UNC
Image Guided Planning of Radiotherapy

Planning in 3D
 Extracting
normal anatomy
 Extracting tumor
 Planning beam poses
MIDAG@UNC
Object Representation in
Medical Image Analysis
 Registration
(find geometric transformation
that brings two images into alignment)
 Radiotherapy
 Fuse
multimodality images (3D/3D) for planning
 Verify patient placement (3D/2D)
 Surgery
 Fuse
multimodality images (3D/3D or 2D) for planning
 Fuse preoperative (3D) & intraoperative (2D) images
 Radiology
 Fuse
multimodality images (3D/3D) for diagnosis
MIDAG@UNC
Object Representation in
Medical Image Analysis
 Shape
& Volume Measurement
 Make
physical measurement
 Radiotherapy

Measure effect of therapy on tumor
 Radiology,

 Find
Neurosciences
Use measurement in science of object development
how probable an object is
 Radiology,
Neurosciences
Use measurement as quantitative input to diagnosis
 Use measurement in science of object development

 Use
as prior in object extraction
 E.g.,
extract the kidney shaped object
MIDAG@UNC
Object Shape & Volume Measurement:
Neurofibromatosis
(Gerig, Greenwood)
Infant Ventricle from 3D
U/S (Gerig, Gilmore)
MIDAG@UNC
Object Extraction (Segmentation)
 Approach 1: preanalyze, then fit to model
 Neurosurgery (MR Angiogram), Radiology (CT)

Vessels, ribs, bronchi, bowel via tube skeletons
 Cardiology
(3D Ultrasound)
Geometry via clouds of medial atoms
 Fit appropriately labeled clouds to 3D LV model

 Cardiac
Nuclear Medicine (2D Gated Blood Pool Cine)
Extract LV, with previous frame providing model
 Extraction via deformable m-rep model
 Shape from extracted LV; analyze shape series

 Surgery,

Radiation Oncology (Multimodality MRI)
Extract tumor, using local shape characteristics
MIDAG@UNC
Extracting Trees of Vessels via
Skeletons (Aylward, Bullitt)
MIDAG@UNC
Presenting Ribs via Tube Skeletons
(Aylward)
MIDAG@UNC
Presenting Bronchi and Lung Vessels
via Tube Skeletons (Aylward)
MIDAG@UNC
Presenting Small Bowel via Tube
Skeletons (Aylward)
MIDAG@UNC
Presenting Blood Vessels Supplying a
Tumor for Embolization (Bullitt)
Full tree, 2D
Subtree, 2D
3D, from 2 poses
MIDAG@UNC
Heart Model (G. Stetten)
cap
cylinder
myocardium
epicardium
left ventricle
slab
mitral valve
left atrium
MIDAG@UNC
Statistical Analysis of Medial
Atom Clouds (G. Stetten)
sphere
cylinder
slab
MIDAG@UNC
LV Tube Identified by Medial Atom
Statistical Analysis (G. Stetten)
sphere
slab
cylinder
MIDAG@UNC
Mitral Valve Slab Identified by Medial Atom
Statistical Analysis (G. Stetten)
sphere
slab
cylinder
MIDAG@UNC
Automatic LV Extraction via Mitral
Valve/LV Tube Axis (G. Stetten)
MIDAG@UNC
Gated Blood Pool Cardiac LV
Cine Shape Analysis (G. Clary)
Example sequence
4-sided medial elliptical analysis
MIDAG@UNC
Object Extraction (Segmentation)
 Approach
2: deform model to optimize reward
for image match + reward for shape normality
 Radiation
Oncology (CT or MRI)
 Abdominal,
pelvic organs
 Deform m-reps model
 Neurosciences
(MRI or 3D Ultrasound)
 Internal
brain structures
 Spherical harmonics boundary model
 Deformable m-reps model
 Neurosurgery
(CT)
 Vertebrae
MIDAG@UNC
M- Reps for Medical Image Object
Extraction and Presentation (Chen,
Thall)
MIDAG@UNC
Displacements from
Figurally Implied Boundary
Boundary implied by figural model
Boundary after displacements
MIDAG@UNC
Vertebral M-reps Model
MIDAG@UNC
Vertebral M-reps Model:
Spinous Process Figure
MIDAG@UNC
Cerebral Ventricle M-reps Model
MIDAG@UNC
Extraction with Object Shape as a
Prior
Brain structures (Gerig)
MIDAG@UNC
Registration
 Registration
(find geometric transformation
that brings two images into alignment)
 Radiotherapy
 Fuse
multimodality images (3D/3D) for planning
 Verify patient placement (3D/2D)
 Surgery
 Fuse
multimodality images (3D/3D or 2D) for planning
 Fuse preoperative (3D) & intraoperative (2D) images
 Radiology
 Fuse
multimodality images (3D/3D) for diagnosis
MIDAG@UNC
Image Guided Delivery of Radiotherapy

Patient placement
 Verification
of plan via portal image
 Calculation of new treatment pose
MIDAG@UNC
Finding Treatment Pose from
Portal Radiograph and Planning DRR
Patient
Setup
Planning
CT Scan
Planning pose
Planning DRR
Candidate poses
Treatment pose
Candidate DRRs
Portal Image
MIDAG@UNC
Medial Net Shape Models
Medial net
Medial nets, positions only
MIDAG@UNC
Image Match Measurment of M-rep
MIDAG@UNC
Registration Using Lung Medial Object
Model : Reference Radiograph (Levine)
Medial net
Medial nets, positions only
MIDAG@UNC
Radiograph/Portal Image Registration (Levine)
Intensity Matching Relative to Medial Model
Medial net
MIDAG@UNC
Shape & Volume Measurement
 Find
how probable an object is
 Training
images; Principal components
 Global vs. global and local
 Correspondence
Hippocampi (Gerig)
MIDAG@UNC
Modes of Global Deformation
Training set:
Mode 1:
x = xmean + b1p1
Mode 2:
x = xmean + b2p2
Mode 3:
x = xmean + b3p3
MIDAG@UNC
Shape & Volume Measurement
 Shape
Measurement
 Modes
of shape variation across patients
 Measurement = amount of each mode
Hippocampi (Gerig)
MIDAG@UNC
Multiscale Medial Model

From larger scale medial net,
interpolate smaller scale medial net and
represent medial displacements
b.
MIDAG@UNC
Summary: What shape
representation is for in
medicine

Analysis from images
 Extract
the “anatomic object”-shaped object
 Register based on the objects
 Diagnose based on shape and volume

Medical science via shape
 Shape
and biology
 Shape-based diagnostic approaches
 Shape-based therapy planning and delivery
approaches
MIDAG@UNC
Shape Sciences
 Medicine
 Biology
 Geometry
 Statistics
 Image Analysis
 Computer
Graphics
MIDAG@UNC
The End
MIDAG@UNC
Options for Primitives

Space: xi for grid elements

Landmarks: xi described by local geometry

Boundary: (xi ,normali) spaced along boundary

Figural: nets of diatoms sampling figures
MIDAG@UNC
Figural Models

Figures: successive medial involution







o
Main figure
Protrusions
Indentations
Separate figures
Hierarchy of figures

o
Relative position
Relative width
Relative orientation
o
o
o
o
o
o
o
o
MIDAG@UNC
Figural Models
with Boundary Deviations
o
 Hypothesis
o
 At
a global level, a figural
model is the most intuitive
o
o
o
o
o
 At
a local level, boundary
deviations are most intuitive
o
o
o
MIDAG@UNC
Medial Atoms

Imply boundary segments
with tolerance
b
rR(- )b


x
rR( )b
Similarity transform equivariant

Zoom invariance implies width-proportionality of
 tolerance of implied boundary
 boundary curvature distribution
 spacing along net
 interrogation aperture for image
MIDAG@UNC
Need for Special End Primitives
 Represent
 non-blobby
objects
 angulated edges, corners, creases
 still allow rounded edges , corners, creases
 allow bent edges
 But
 Avoid
infinitely fine medial sampling
 Maintain tangency, symmetry principles
MIDAG@UNC
Coarse-to-fine representation
 For
each of three levels
 Figural
hierarchy
 For each figure,
net chain, successively smaller
tolerance
 For each net tile,
boundary displacement chain
MIDAG@UNC
Multiscale Medial Model
 From larger scale medial net
 Coarsely sampled
 Smooother figurally implied boundary
 Larger tolerance
 Interpolate smaller scale medial net
 Finer sampled
 More detail in figurally implied boundary
 Smaller tolerance
 Represent
medial displacements
MIDAG@UNC
Multiscale Medial/Boundary Model
 From medial net
 Coarsely sampled, smoother implied
boundary
 Larger tolerance
 Represent
boundary displacements
along implied normals
 Finer
sampled, more detail in boundary
 Smaller tolerance
MIDAG@UNC
Shape Repres’n in Image Analysis
 Segmentation
Find
the most probable deformed
mean model, given the image
Probability
involves
 Probability
of the deformed model
 Probability of the image, given the
deformed model
MIDAG@UNC
Medialness: medial strength of
a medial primitive in an image
Probability of image | deformed model
 Sum of boundariness values





at implied boundary positions
in implied normal directions
with apertures proportional to
tolerance
b
rR(- )b

x
rR( )b
Boundariness value

Intensity profile distance from mean (at scale)
MIDAG@UNC
Shape Rep’n in Image Analysis
 Segmentation
 Find the most probable deformed mean
model, given the image
 Registration
 Find
the most probable deformation, given the
image
 Shape Measurement
 Find how probable a deformed model is
MIDAG@UNC
Object Shape
Representations for Medicine to Manufacturing
 Figural
models, at successive levels of
tolerance
 Boundary displacements
 Work
in progress
 Segmentation
and registration tools
 Statistical analysis of object populations
 CAD tools, incl. direct rendering
…
MIDAG@UNC