Download Peering into the Future Through the Looking Glass of Artificial

Eliot Siegel, MD, FSIIM, FACR
Professor of Radiology
University of Maryland School of Medicine
Chief Imaging Services, VA Maryland Healthcare System
Dwyer AI Session Outline
• Tanveer F. Syeda-Mahmood, PhD
– Chief Scientist, Medical Sieve
Radiology Grand Challenge
– IBM Almaden Research Center
• Panel Discussion
– Rasu B. Shrestha, MD, MBA, Chief
Innovation Officer, University of
Pittsburgh Medical Center, Executive
Vice President, UPMC Enterprises;
– Khan M. Siddiqui, MD, Co-Founder,
Chief Technical Officer, higi, CoDirector, Center for Biomedical &
Imaging Informatics, Visiting
Associate Professor Radiology, Johns
Hopkins University
Disclaimer
– ACR
– Fuji
– Philips
– AGFA
– Galileo
– RADLogics
– Amalga
– Herman Miller
– Radsite
– Anatomic Travelogue
– IBM
– Redrick/Evolve
– Anthro
– Intel
– RSNA
– Applied Radiology
– Kodak
– Siemens
– ATL
– NIBIB
– SIIM
– Barco
– NIST
– Sonare
– Bracco
– NLM
– Steelcase
– Brightfield
– NCI
– TeraRecon
– Carestream
– Life images
– Topoderm
– Cydar
– McCoy
– Toshiba
– Dejarnette
– McKesson
– Virtual Radiology
– Dell
– Medrad
– Vital Images
– Diagnostic Imaging
– Medscape
– Xybix
– Digital Art Forms
– Merge
– YYESIT
– Dynamic Imaging
– Microsoft
– Zebra
– Eizo
– Montage
– Fovia
– GE
Samuel J. Dwyer, III, PhD,
FSIIM (1932–2008)
• On May 4, 2002 Sam became the ninth SCAR
member to be inducted into the SCAR College of
Fellows.
• Dr. Dwyer received his PhD in Electrical
Engineering at the University of Texas-Austin
specializing in systems and signal processing
• Dr. Dwyer at the time of his retirement was a
Professor of Radiology at the University of
Virginia Health Sciences System
Steve Horii, M.D.
• I knew him to be always ready with a smile or
infectious laugh and with a perpetual gleam in
his eye that spoke of his friendly manner
• There are some who would claim the title of
“PACS Man”, but it is Sam Dwyer who led the
revolution in PACS
• Sam Dwyer was a major pioneer who brought
many of the important advances in technology to
us and helped move concepts from the realm of
engineering to that of healthcare
• I will miss Sam very much, but the strong
memory of him is never further than the PACS
workstation I use every day
R. Gilbert Jost, MD
Past President RSNA and RISC
• If one were to identify a “father of PACS”,
unquestionably it would be Sam Dwyer…
• He is truly a pioneer who has changed the
specialty of radiology for the better in
innumerable ways
Continued Relationship with
Sam
• Sam was very reassuring when we became the
world’s first filmless hospital that our problems
would not be insurmountable and that the time was
right to go filmless
• As this “kid” right out of residency aspiring to the first
filmless hospital, I think I amused Sam
• In subsequent years we got together frequently at
RSNA and he often scribbled messages and
drawings on napkins and handed them to me
• Kept in contact by phone and e-mails and always
enjoyed talking with him
• Makes me wonder what Sam would have thought
about “Artificial Intelligence” and its potential in
diagnostic imaging
Where Are We in 2016 in AI in
Diagnostic Imaging?
• Sam would be surprised that CAD hasn’t
made more progress in diagnostic imaging
• 10’s of thousands of machine learning
algorithms but almost no connection
between the research and clinical
application of these
• Relatively small incremental improvements
in fairly narrowly defined image analysis
algorithms, e.g. mammography CAD, lung
nodule detection, vascular stenosis analysis
Where Are We in 2016?
• Is there a generalized learning
algorithm/program for imaging that could
create a jumpstart to a major advance in
diagnostic imaging?
• Amazing advances in ML and AI in many
domains high publicity
• Lots of companies lately are claiming to
have made that jump
• How much is reality and how much is
hype?
Dedication: to João Louro
11
12
The Economist and Others Are Talking
about the 4th Industrial Revolution Based
on Cyber-Physical Systems
The substitution of machinery for machine
labour” may “render the population
redundant
The discovery of this mighty power” has
come “before we knew how to employ it
rightly”
the
Debate in early 1800s about
industrial revolution in England
Easy to Replace?!
• Andrew Ng, renowned Stanford Professor
and expert on machine learning was quoted
in The Economist this week as saying “a
highly trained and specialized radiologist may
now be in greater danger of being replaced
by a machine than his own executive
assistant: She does so many different things
that I don’t see a machine being able to
automate everything she does any time
soon.”
Ezekiel Emanuel, PhD,
MD, MSc
• Gave keynote presentation at ACR 2016
– Faculty member at the Wharton School and
School of Medicine at University of
Pennsylvania
– Founding chair of the Clinical Center of the
National Institutes of Health
– Former special advisor on health policy for the
Office of Management and Budget.
Ezekiel Emanuel, PhD, MD,
MSc
Keynote ACR 2016
Five Megatrends
•
•
•
•
Decline in the use of hospitals
More outpatient care
More care in patients’ homes
Fewer medical tests
• Machine learning
– “While all of these factors will shape the future landscape,
machine learning will be the most pressing for radiology
– Emanuel called the machine learning “the real
threat to radiology.”
• “The biggest barrier will not be technical but human willingness to
accept machine based diagnoses.”
Major “Inspiration”/”Motivation” for This
Year’s 2016 Sam Dwyer Lecture
• Visiting Professor at Hospital of
University of Pennsylvania
• CEO of well funded and well known
start-up company in medical imaging
space related that he wanted to
(paraphrased) “get rid of the wasted
protoplasm sitting in front of the
workstation that was the radiologist and
replace it with a much better and reliable
and consistent alternative in the next
few months”
Stephen Hawking on AI
• “Success in creating AI would be the biggest event in human
history,” wrote Stephen Hawking in an op-ed, which appeared in
The Independent in 2014.
• “Unfortunately, it might also be the last, unless we learn how to
avoid the risks. In the near term, world militaries are considering
autonomous-weapon systems that can choose and eliminate
targets.” Professor Hawking added in a 2014 interview with BBC,
“humans, limited by slow biological evolution, couldn’t compete and
would be superseded by A.I.”
• Hawking told the BBC: “The primitive forms of artificial intelligence
we already have, have proved very useful. But I think the
development of full artificial intelligence could spell the end of the
human race.”
19
Elon Musk
• Elon Musk has spoken out against artificial intelligence
(AI), declaring it the most serious threat to the survival of
the human race to students from Massachusetts Institute
of Technology (MIT)
• “I think we should be very careful about artificial
intelligence. If I had to guess at what our biggest
existential threat is, it’s probably that. So we need to be
very careful,” said Musk
• “I’m increasingly inclined to think that there should be
some regulatory oversight, maybe at the national and
international level, just to make sure that we don’t do
something very foolish.”
Bill Gates
• Microsoft co-founder Bill Gates has also expressed
concerns about Artificial Intelligence
• During a Q&A session on Reddit in January 2015,
Mr. Gates said, “I am in the camp that is concerned
about super intelligence. First the machines will do a
lot of jobs for us and not be super intelligent. That
should be positive if we manage it well
• A few decades after that though the intelligence is
strong enough to be a concern
• I agree with Elon Musk and some others on this and
don’t understand why some people are not
concerned.”
More Dangerous Than Nuclear Weapons?
• Mr. Hawking recently joined Elon Musk, Steve
Wozniak, and hundreds of others in issuing a letter
unveiled at the International Joint Conference in
Buenos Aires, Argentina
• The letter warns that artificial intelligence can
potentially be more dangerous than nuclear
weapons.
22
Who Is Investing All Those Dollars in
Artificial Intelligence?
Investment in Artificial
Intelligence
• Ironically, given Elon Musk and Sam Altman’s
concern that artificial intelligence will take over
the world, the two entrepreneurs are putting
more than a billion dollars into a not-for-profit
company that will maximize the power of AI—
and then share it with anyone who wants it
• In an interview with Steven Levy of
Backchannel about Open AI, Altman said they
expect this decades-long project to surpass
human intelligence
• But they believe that any risks will be mitigated
because the technology will be “usable by
everyone instead of usable by, say, just
Google.”
Many Many AIs
and Dr. Evil
• They were asked whether their plan to freely share
this technology would actually empower bad
actors, if they would end up giving state-of-the-art
AI to the Dr. Evils of the world. But they played
down this risk
• They feel that the power of the many will outweigh
the power of the few. “Just like humans protect
against Dr. Evil by the fact that most humans are
good, and the collective force of humanity can
contain the bad elements,” said Altman, “we think
its far more likely that many, many AIs, will work to
stop the occasional bad actors.”
25
AI/Machine Learning Basic Terms
Deep Learning Falls Within
Machine Learning Within AI
Artificial
Intelligence
• Basically an umbrella term for a variety of
applications and techniques
• Artificial intelligence refers to "a broad set of
methods, algorithms and technologies that make
software 'smart' in a way that may seem humanlike to an outside observer”
» Lynne Parker, director of the division of Information and
Intelligent Systems for the National Science Foundation
• John McCarthy, who coined the term “Artificial
Intelligence” in 1956, complained that “as soon
as it works, no one calls it AI anymore.”
Artificial Intelligence
• Machine learning, computer vision,
natural language processing, robotics
and related topics are all part of A.I.
• Also referred to as “machine intelligence”
or “computational intelligence”
• Can distinguish different types of AI
• When will AI Arrive?
– It’s here already!!!
Is There A General Equation for
Winning at AI?
• The action-value function is the maximum sum of
rewards rt discounted by γ at time step t,
achievable by a behavior policy Π=P(a|s), after
making an observation (s) and taking an action (a)
• Can be optimized using a Deep convolutional
neural network
• Key to winning at Atari Video games
• Key to “happiness”?
• Key to LIFE?
Where is AI? Everywhere. My Monday
Morning Diary
• Wake up by iphone, One Dance/Drake • Arrive at work at VA Hospital
• Check indoor temperature on Nest
• Big stack of papers on desk to be
signed-rummage through drawers to
• Amazon Echo checks out weather and find pen and move papers from one
traffic on the way to work and turns on
side of the desk to the other
the lights
• Take 10 minutes to sign into EMR to
• Google Now says flying to Portland
check consults after waiting about
tomorrow with itinerary
• Take another 8 minutes to sign into
• Set temperature in the Car on iphone
PACS
• Get read for work while Siri plays latest • Take 10 minutes to play messages on
unread messages and e-mails
phone machine
• Car displays today’s schedule from
• Grab stack of paper requisitions to
Google calendar and goes on autopilot protocol
and does 95% of driving to work
autonomously
Artificial Intelligence (Narrow)
• Also referred to as Weak AI
• AI that specializes in one area
• There’s AI that can beat the world
chess champion in chess, but that’s
the only thing it does
– Speech recognition
– Translation
– Self-driving cars
– Siri, Alexa, Cortana, Google Now
Artificial General Intelligence
• Sometimes referred to as Strong AI, or
Human-Level AI
• Computer that is as smart as a human
across the board—a machine that can
perform any intellectual task that a human
being can
• Creating AGI is a much harder task than
creating ANI, and we are nowhere near
close to it
Artificial General Intelligence
(AGI)
• Professor Linda Gottfredson describes
intelligence as “a very general mental
capability that, among other things,
involves the ability to:
– Reason
– Plan
– Solve problems
– Think abstractly
– Comprehend complex ideas
– Learn quickly
– Learn from experience”
When Will AGI Arrive?
• A study, conducted recently by author
James Barrat at Ben Goertzel’s annual AGI
Conference asked when participants
thought AGI would be achieved—by 2030,
by 2050, by 2100, after 2100, or never. The
results:
• By 2030: 42% of respondents
• By 2050: 25%
• By 2100: 20%
• After 2100: 10%
• Never: 2%
Artificial
Superintelligence
• Oxford philosopher and leading AI
thinker and author Nick Bostrom
defines super-intelligence as “an
intellect that is much smarter than
the best human brains in
practically every field, including
scientific creativity, general
wisdom and social skills.”
Machine Learning
• Also blanket term that covers multiple
technologies
• Doesn’t necessarily have to actually
“learn” as we think of it and doesn’t
necessarily provide feedback over
time just refers to a class of statistical
techniques to characterize, discover,
classify data
• Vast majority of these have been
around for many years/decades
Machine Learning
• As a part of A.I., machine learning refers to
a wide variety of algorithms and
methodologies that can also enable
software to improve its performance over
time as it obtains more data
• "Fundamentally, all of machine learning is
about recognizing trends from data or
recognizing the categories that the data fit in
so that when the software is presented with
new data, it can make proper predictions,"
(Parker)
Neural Networks
• Neural networks are a type of
machine learning, and deep learning
refers to one particular kind
• Neural networks -- also known as
"artificial" neural networks -- are one
type of machine learning that's loosely
based on how neurons work in the
brain, though "the actual similarity is
very minor”
Neural Networks
• There are many kinds of neural networks, but in
general they consist of systems of nodes with
weighted interconnections among them
• Nodes, also known as "neurons," are arranged
in multiple layers, including an input layer where
the data is fed into the system; an output layer
where the answer is given; and one or more
hidden layers, which is where the learning takes
place
• Typically, neural networks learn by updating the
weights of their interconnections
Examples Neural Network Types
Types of Neural Networks:
Autoencoder
• Autoencoder is a simple 3-layer neural network where
output units are directly connected back to input units.
• Relatively simple and intuitive
Restricted Boltzman Machine
• Intuition behind RBMs is that there are some
visible random variables (e.g. film reviews from
different users) and some hidden variables (like
film genres or other internal features), and the task
of training is to find out how these two sets of
variables are actually connected to each other
Convolutional
Neural Networks
• Like Autoencoders and RBMs- translate many
low-level features (e.g. user reviews or image
pixels) to a compressed high-level
representation (e.g. film genres or edges) - but
now weights are learned only from neurons that
are spatially close to each other.
• CNNs are very specifically optimal for image
recognition. Most of the top-level algorithms in
image recognition are somehow based on
CNNs today
Purpose of These Neural Networks is
Dimensionality Reduction
• Autoencoders and RBMs both take a vector in nndimensional space they translate it into an mmdimensional one, trying to keep as much important
information as possible and, at the same time, remove
noise
• If training of autoencoder/RBM was successful, each
element of resulting vector (i.e. each hidden unit)
represents something important about the object - shape
of an eyebrow in an image, genre of a film, field of study
in scientific article, etc.
• You take lots of noisy data as an input and produce
much less data in a much more efficient representation
Neural Networks Used for
Pretraining then Other Classifier
Used
• None of models mentioned here work as
classification algorithms per se
• Instead, they are used for pre-training learning transformations from low-level and
hard-to-consume representation (like
pixels) to a high-level one
• Once deep (or maybe not that deep)
network is pretrained, input vectors are
transformed to a better representation and
resulting vectors are finally passed to real
classifier (such as SVM or logistic
regression)
Deep Learning
Neural Networks
Deep Learning
• Deep learning refers to what's sometimes
called a "deep neural network," or one that
includes a large system of neurons
arranged in several hidden layers
– A "shallow" neural network, by contrast, will
typically have just one or two hidden layers.
• The idea behind deep learning is not new,
but it has been popularized more recently
because we now have lots of data and fast
processors that can achieve successful
results on hard problems
Commonly Used Machine
Learning Techniques
• Regression techniques
• Neural networks
• Support vector machines
• Decision trees
• Bayesian belief networks
• k-nearest neighbors
• Self-organizing maps
• Case-based reasoning
• Instance-based learning
Machine Learning Vs. Data Mining
• Machine learning focuses on
prediction, based on known
properties learned from the
training data.
• Data mining focuses on the
discovery of (previously) unknown
properties in the data
Machine Learning vs.
Optimization
•Optimization algorithms can
minimize the loss on a
training set
•Machine learning is
concerned with minimizing
the loss on unseen samples
Machine Learning and Statistics
and “Statistical Learning”
• Machine learning and statistics are closely related
fields and machine learning can be considered a
statistical technique
• Leo Breiman distinguished two statistical modeling
paradigms: data model and algorithmic model,
wherein 'algorithmic model' means more or less
the machine learning algorithms like Random
forest
• Some statisticians have adopted methods from
machine learning, leading to a combined field that
they call statistical learning
What is Deep Learning?
• DL consists of multiple hidden layers in an
artificial neural network
• This approach tries to model the way the
human brain processes light and sound into
vision and hearing
• Two very successful applications of deep
learning are computer vision and speech
recognition
• Falling hardware prices and the development
of GPUs for personal use in the last few years
have contributed to the development of the
concept of Deep Learning (DL)
Deep Learning vs. Machine
Learning
56
ImageNet
Large Scale Image Recognition
Challenge Started in 2010
• Computers have always had trouble
identifying objects in real images so it
is not hard to believe that the winners
of these competitions have always
performed poorly compared to
humans.
• But all that changed in 2012 when a
team from the University of Toronto in
Canada entered an algorithm called
SuperVision, which wiped the floor
with the opposition.
SuperVision
• SuperVision, for example,
consists of some 650,000
neurons arranged in five
convolutional layers
• It has around 60 million
parameters that must be finetuned during the learning process
to recognize objects in particular
categories.
61
Speech Recognition Deep
Learning Breakthrough
Human Vision: The Hardest
Task for Computers?
Introduced by Alan Turing in his 1950
paper “Computing Machinery and
Intelligence”
Opens with the words “I propose to
consider the question, ‘Can machines
think?”
Asks whether a computer could fool a
human being in another room into thinking
it was a human being
Modified Dr. Watson Turing Test might
ask: Can a computer fool a human being
into thinking it was a doctor?
What’s Wrong with this Picture?
Ultimate Challenge: Medical
Imaging
Scientific American June 2011
Testing for Consciousness
Alternative to Turning Test
Highlights for Kids “What’s Wrong
with this Picture?”
Christof Koch and Giulio Tononi
Imaging May Be Ultimate/Future
Frontier For “AI” Software
Machine Learning Algorithms:
Like Standards, So Many to Choose From!
Can We Apply Those Incredible
Advances in Object Recognition
to Diagnostic Radiology?
• These image challenges have used 24 big “RGB”
color images with no experience with gray scale
imaging in medicine
• They can identify a chair but can’t tell if it’s
–
–
–
–
–
–
Broken
Something is missing
Something extra is there
Comfortable
Beautiful or ugly
Dirty or clean
• Black box – Can’t explain why something is
identified as abnormal
• Adrenal challenge 5th Grader– Need to know
anatomy
“Magic” Aspect of
Deep Learning
• One major challenge is that we don’t understand
what’s inside black box of deep learning when it
solves a visual recognition challenge
• Don’t need deep learning for Tic Tac Toe or
Checkers or even chess because we can use
combination of brute force to look at every possible
move (chess out to 20 to 30 or more moves and
further at the end game)
• But game like Go or playing video games, can’t do
brute force but can learn by trial and error even
though black box without understanding of why,
like magic
• No general purpose learning system for
diagnostic imaging like we train our
residents
• Our eyes and brains have evolved to detect
patterns and our knowledge of medicine,
physiology, a priori likelihood of disease
and recognition of trends evolved over
millions of years
Challenges for Machine Learning
Algorithms
• Which to choose from?
• How do evaluate different machine learning algorithms
and determine which is most efficient for a particular
problem?
• Black box?
• How to optimize parameters?
– Practical Beyesian Optimization Machine Learning
• Computational Time
• Very specific MLA’s do a good job at different tasks which
makes it difficult to select a single one as a generalized
deep AI approach for image analysis or for data analysis
• Problems with High Dimensional Datasets like electronic
medical record requires different approach
Applications of “Machine Learning” in
Medical Imaging
These Machine Learning Techniques Have Been Utilized in
Imaging for Decades With Tens of Thousands of Published Papers
10,000s of Narrow Machine Learning Applications
in Medicine
Challenge is Binding these together and can one
develop general learning theory?
• Fracture detection
• Brain hemorrhage
• Mammography
• MS diagnosis and quantification
• Bone age determination
• Lung nodule detection
• Liver mass determination
• Meniscal tear
• Brain segmentation and diagnosis
• Bone mineral density on CT
• Carotid stenosis evaluation
• Coronary Artery stenosis evaluation
• Cardiac function evaluation
Eliot Siegel, M.D.
Prof. and Vice Chair University of Maryland
Chief Imaging VA Maryland Healthcare System
Black Box of CAD
• The “black box” nature of CAD is seen as a
substantial issue by many radiologists
• If my residents and fellows told me they
thought this right upper nodule was cancer
and I asked why and they wouldn’t say why
or how confident they were, I’d:
– Be less confident
– Be suspicious about their analysis
– Be frustrated
What Made You
Circle the
Lesion?
• Lesion size
• Lesion morphology (shape: smooth,
spiculated)
• Density distribution (solid, ground glass,
partially calcified)
• Location (subpleural, which lobe)
• Connectedness (is it connected to vessels or
other structures?)
Level of Confidence and Quality
of Evidence
• What was your level of confidence in the magic box formula of the
above that made you circle it?
– Did it have to meet size, morphology, density and
connectedness or even location characteristics
– What database did you use to determine level of
suspicious of cancer and how many cases were in it
or was it based on expert opinion?
• 10
• 100
• 1000
• 10,000
• More?
Left Upper Lobe Lung Nodule
CAD is Pretty Sure It’s There
CAD is Not So Confident
Where Are We Today With
Clinical Use of CADe?
• Mammography is far and away the most
utilized application
• But what do radiologists really think of
Mammography CAD?
Is CAD gaining momentum in
clinical practice?
– It seems to be happening too slowly, more
slowly so than most of us had anticipated
– In cases (unlike mammography) where there is
no reimbursement for CAD, the radiologists and
practices are feeling that their margins are low
enough and there is major pressure related to
decreased reimbursement and the impression
that reimbursement will continue to drop
• Difficult to make business case for added
expenditure for CAD to radiologists
Is CAD Gaining Momentum In
Clinical Practice?
– There is much skepticism among my colleagues
about the added value of CAD and many only
use it for mammography because of the
reimbursement model
– Colleagues will not pay any significant amount
for say, CAD lung nodule detection for chest
radiography even with a hypothetical scenario of
a 10 or even 20% increase in sensitivity
– I believe that they would pay more for
something that increased their efficiency and
productivity than their accuracy
MATERIALS AND
METHODS:
• Separate links to an online survey were
posted on the website of the Society of
Breast Imaging and circulated to subscribers
of Diagnostic Imaging.com, in order to
evaluate opinions regarding CAD use and its
underlying legal issues
RESULTS: Use and Reliance on CAD?
• 89% indicated they always use CAD
when reading screening mammograms
• 4% indicated that they rarely or never
use CAD
Use and Reliance on CAD
and Reimbursement
• However the extent to which clinicians are
relying on CAD to provide an accurate
diagnosis is split
– 2% indicated that they always rely on CAD to
provide an accurate diagnosis
– 49% indicated they sometimes rely on CAD
– 49% clinicians rarely or never rely on CAD
– It is likely that the mismatch between use of and
reliance on CAD relates to the reimbursements
($12, or $1000 per approximately 83 cases or
$2,400 per day for 200 screening mammograms)
radiologists receive when using CAD
RESULTS: Use and Reliance on CAD?
§ Most radiologists have not changed a
read based on the results of CAD
§ Only 2% indicated they alters their
opinion after CAD
§ 36% sometimes change interpretation
based on CAD
§ 61.7% rarely or ever change their
interpretation based on CAD
Use and Reliance on CAD
• 15% found that CAD was often helpful
• 49% considered it sometimes helpful 36%
considered it rarely or never helpful
What Do I Need from
Next Generation CAD
Clinically?
• Improve efficiency/productivity
• Increases my accuracy/reliability
without compromising efficiency
• Affordable
• Increases my confidence
• Allows me to measure things I couldn’t
measure otherwise such as liver or
pulmonary “texture”
• Provide Imaging “Physical Exam”
Next Generation of CAD
• The next generation of CAD will reflect the trend
toward big data and personalized medicine and
shift away from the current second reader
approach and toward one in which CAD
algorithms increasingly serve as visualization
and image measurement/annotation and
quantification tools
– Examples of probability maps rather than just binary
yes and no and FDA requirements shaped the second
reader
– Tracking lesions over time
– Highlighting certain types of findings to draw attention
to the reader
CAD Tools Requirements and
Challenges
• CAD applications must be able to be integrated into the
image acquisition, display and interpretation workflow
• They will not be adopted if they constrain the throughput of
the radiologist
• Need high level of accuracy in a single patient, need to
more than just demonstrate efficacy in separating two
groups
• Commercialization and U.S. Food and Drug Administration
(FDA) clearance is a big hurdle and needs to be revisited
Next Generation CAD
Apps Store for CAD Algorithms?
• Want to be able to utilize all of these on a
single platform, e.g. using API specified by
DICOM working group 23
• Would like to see business ecosystem such
as GRID that could allow users to have a
payment model for these so you could
download algorithm on the fly or send
images up to a web service or could get
consensus from multiple CAD programs
Israeli StartUp
• Platform to create and integrate a
variety of algorithms to test against 12
million anonymized, indexed and
catalogued imaging studies
• Supports multiple coding languages and libraries,
including Machine Learning Convolutional Network
libraries such as Torch, Cafe and Theano, image
processing libraries
• All work is saved and projects can be collaborated
on by several users
• In addition, provide high end, dedicated GPUs and
CPUs to run algorithms
Second Start-up Company Creates
Preliminary Report for Chest CT
IBM Medical Sieve
Recommendations for CAD
• I believe the FDA has often limited the
challenges to CAD as a second reader
rather than as a tool that can be toggled on
and off
– Would like to see highlighted images like a spell
checker that could also color code probability
that a finding is real/confidence of the CAD
algorithm
– I’d like to see CAD to do image recognition
before a study is reviewed as screening for
things such as rib fractures, compression spine
fractures, pneumothorax, etc. the equivalent of
an imaging physical exam
Applying Human Vision Research to CAD
Adrenal Challenge
• Challenge:
–Can any CAD program find the adrenals
as well as you could teach an 8 year old
child in ten minutes?
–I have never seen anyone successfully
tackle the problem of finding the
adrenals
–This would have substantial value
MACHINE
LEARNING AND AI
FOR LUNG
NODULE
SCREENING
Eliot Siegel, MD, FACR
Prof/Vice Chair IS Univ.
Maryland
Five Levels IT Decision Support
for Lung Nodule Screening
1. Radiologist interprets nodule based on
clinical experience and makes
measurement and report
2. Radiologist interprets study based on LungRADS criteria and reports score after
looking up information on paper or online
3. Radiologist has Lung-RADS criteria
available and EMR/PACS automatically
pulls up required information to make it
easy and in one place
Five Levels IT Decision Support for
Lung Nodule Screening With Mr.
Akami Example
4. PACS brings up lung rads criteria in the context
of interpretation automatically with ACR Assist
with data automatically classified according to
template
5. Finally, Automatic click on nodule initiates
search of database such as NLST which then
finds similar patients, categorizes based on risk
of a specific cohort based on detailed nodule
and patient information, utilizes a priori
probability based on PLCO and combines those
for patient specific probability of disease and
then also maps out to LungRads
ACR Assist:
Future Automated Reporting Using Template Directly to
Report and Registry
Standa
rds
Commi
ttee
Structured Content
Structured Input
Evidence-Based XML Encoding
Algorithms
Actionable ReportsCritical Test Results Mgmt
Structured Output
Integrating
Workflow
Registries
Advanced Decision Support in Action:
Your Next Door Neighbor Mr. Akami
• Your next door neighbor and friend,
•
•
•
•
Mr. Akami, a 62 year old native
Hawaiian smoker with COPD who
gets admitted for an elective
Bunionectomy
7 mm spiculated soft tissue density
left lower lobe nodule is discovered
on “routine” pre-op exam and
confirmed on CT with no other
abnormalities
What is the likelihood that it is
malignant?
How should this nodule be followed
up?
Do we have tools at the workstation
while reporting to help apply the
ACR LungRads criteria to help out
the radiologist?
National Lung Screening Trial Dataset and Decision Support Project
Taking Personalization to the Next Level Beyond LungRads
Reporting Tool
• Can we personalize the ACR Lung Rads criteria
•
•
•
•
•
using data from the National Lung Screening Trial?
Could the criteria for follow-up be refined and
personalized more than high risk smoker vs. lower
risk patient based on:
Geographic location?
Patient age/sex?
Characteristics of nodule e.g. shape
(spiculated or smoothly rounded), containing
calcification?
Presence of additional nodules?
National Lung Screening Trial
(NLST)
26,721 participants
32,289 nodules
Searching By Cohort Match
5% of Nodules for Males 60 to 65
that were 5-7mm were Malignant
N/A 0
<1 1, 2
%
13
2%
4A
515%
>15% 4B
Mr. Akami category 3 suggesting
1-2% prob. of malignancy with 6
6
0
2
9
5
4
8
1
Next Phases of NLST Analysis
• Problem with cohort analysis is that cohorts get small very
quickly with increasing number of variables
• Linear regression analysis
• Look up formula for linear regression from literature
• More advanced multi-regression analysis would come
closer to being the best fit
• Bayesian Approach
• Machine learning algorithm even better such as
• Support Vector Machine
• Many other machine learning possibilities
• Deep Learning
Now Add Nodule Shape Matching To
Further Personalize and Refine Accuracy
Next Step: Pixel Analysis NLST Images
Statistics Features
Shape Features
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
1.
2.
3.
4.
5.
Center Of Gravity
Histogram
Kurtosis
Maximum
Maximum Index
Mean
Median
Minimum
Minimum Index
Skewness
Standard Deviation
Sum
Variance
Weighted Elongation
Weighted Flatness
Weighted Principal Axes
Weighted Principal Moments
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
Bounding Box
Centroid
Elongation
Equivalent Ellipsoid Diameter
Equivalent Spherical
Perimeter
Equivalent Spherical Radius
Feret Diameter
Flatness
Number Of Pixels
Number Of Pixels On Border
Perimeter
Perimeter On Border
Perimeter On Border Ratio
Physical Size
Principal Axes
Principal Moments
Roundness
Extract Nodules and Then Apply One of A
Wide Variety of Machine Learning
Algorithms Especially Convolutional
Neural Network
PLCO Dataset
PLCO Dataset
“Instant Research”
Personalized Clinical Care
122
PLCO Participants Who Qualify for NLST
123
Creating Local/Regional Databases from
Clinical Data
• Would also like to be able to collect data at the University
of Maryland, within the Department of Veterans Affairs
Hospitals in Maryland and then nationally that could
establish a similar database
• Then could provide report that gave reference database
such as NLST with likelihood of malignancy and also gave
local reference to a specific population and then taking
into account PLCO data
NLST and PLCO Next Steps
• Huge implications for screening, e.g. reduce cost
from over $200,000 per life saved for smokers over
50 years old to a lower cost for a higher risk cohort
for screening studies
• PLCO use has major implications for Bayesian pretest probability data to assist in diagnosis
• Working with multiple vendors, demonstrating
ability to incorporate this into the workflow with
ability to “click” on nodule and then have
automated lesion characterization, lookup from
EMR and then access reference database
“service” to get information about likelihood of
malignancy
126
• Would like to incorporate these data into
routine applications such as CAD software
that could take a priori probability of
disease to help to CADx in addition to
current CADe, e.g. if patient had prior
breast cancer CAD should “realize” odds
of another breast cancer higher and adjust
accordingly
• Would like to create on the fly
statistical/machine learning models rather
than just finding similar patients in
databases such as NLST or PLCO
127
• IT is becoming increasingly critical to the success of today’s
practice of radiology and is especially critical as we move to
implement the complex process that is associated with Lung
Cancer Screening
• Clinical decision support tools are evolving from the current
state of the art to next generation and beyond systems that
will allow us to take care of patients such as Mr. Akami in an
increasingly safe and effective manner
• This will allow us to maximize the likelihood that our CT
screening studies will save lives and reduce morbidity
associated with lung cancer
128
129
130
Professor University of Maryland School of Medicine
Chief Imaging VAMHCS
Adjunct Professor Computer Science UMBC
Board Scientific Counselors National Library of Medicine
What is ORiGAMI ?
An Artificial Intelligence Workflow for
Discovering Novel Associations in
Massive Medical Knowledge Graphs
ORiGAMI under the hood
ORiGAMI: Case “Patient as Art”
This patient, like the artist who made her famous, was a “cripple and an outsider,” though she was
not always so. She began life as a small, blond-haired girl with a “silver giggle,” who seemed no
different from other children. However, by the time she reached the age of three, she was walking
on the outside of her feet with an odd gait. Even so, she was a bright, fiercely determined child, who
stomped around ignoring her disability as it gradually increased in severity. By the time she was 13,
she stumbled and fell frequently, though with a mind so “bright, curious and hungry,” her teacher
had hopes that she too would one day be a teacher. The patient, if not beautiful at age 19, was slim
and handsome enough to attract a suitor, who claimed that during their brief courtship, she could do
anything–“row a boat, climb a tree, harness a horse, and drive a carriage.” Her letters at that time,
however, told a different story, one involving a series of “bad falls.” Her one and only suitor vanished
from her life as suddenly as he had appeared. The patient’s balance soon worsened to the point
that it was unsafe for her to look up without having a firm grip on something for steadiness.
Although she was still able to walk, her crablike gait forced her to use the entire width of the road
when ambulating. Her mother made her kneepads to wear under her skirt as protection against her
many falls. Her hands, as yet unaffected, were capable of the intricate work of a talented
seamstress. By the time she reached 26, the patient could walk only three or four steps without
assistance, and her hands had become so misshaped and unsteady she had to her wrists, elbows,
and knees to do those things formerly done with her hands. Offers of help were gently but firmly
refused. By the end of her fifth decade, she had lost the ability to stand and resorted to crawling to
get where she wanted to go. Her mind continued to be as sharp as ever. No neurological disorders
are known to have affected other members of the patient’s family. Her father was a Swedish sailor
with a disabling arthritis, who died at age 72 of unknown cause. Her mother developed kidney
disease in her 40s and died edematous at age 68 of either renal failure or congestive heart failure.
There were three brothers, one who died in his 80s of metastatic bone cancer. The medical
histories of the other two are unknown. The patient was evaluated medically just once, when she
was 26, at the Boston City Hospital. After a week of observation and tests failed to produce a
diagnosis, she was told “to just go on living as [she] had always done. When the patient was 56,
she developed a severe illness thought to have been pneumonia. One evening, while recuperating,
she sat with one leg stretched out beneath a stove and fell asleep. When she awoke, the heat from
the fire had seared the flesh from her withered leg. The third-degree burn healed slowly in response
to repeated application of cod liver oil. At age 74, the patient finally consented to the use of a
wheelchair and died shortly thereafter.
ORiGAMI: Natural Language Processing
This patient, like the artist who made her famous, was a “cripple and an outsider,” though she was
not always so. She began life as a small, blond-haired girl with a “silver giggle,” who seemed no
different from other children. However, by the time she reached the age of three, she was walking
on the outside of her feet with an odd gait. Even so, she was a bright, fiercely determined child, who
stomped around ignoring her disability as it gradually increased in severity. By the time she was 13,
she stumbled and fell frequently, though with a mind so “bright, curious and hungry,” her teacher
had hopes that she too would one day be a teacher. The patient, if not beautiful at age 19, was slim
and handsome enough to attract a suitor, who claimed that during their brief courtship, she could do
anything–“row a boat, climb a tree, harness a horse, and drive a carriage.” Her letters at that time,
however, told a different story, one involving a series of “bad falls.” Her one and only suitor vanished
from her life as suddenly as he had appeared. The patient’s balance soon worsened to the point
that it was unsafe for her to look up without having a firm grip on something for steadiness.
Although she was still able to walk, her crablike gait forced her to use the entire width of the road
when ambulating. Her mother made her kneepads to wear under her skirt as protection against her
many falls. Her hands, as yet unaffected, were capable of the intricate work of a talented
seamstress. By the time she reached 26, the patient could walk only three or four steps without
assistance, and her hands had become so misshaped and unsteady she had to use her wrists,
elbows, and knees to do those things formerly done with her hands. Offers of help were gently but
firmly refused. By the end of her fifth decade, she had lost the ability to stand and resorted to
crawling to get where she wanted to go. Her mind continued to be as sharp as ever. No
neurological disorders are known to have affected other members of the patient’s family. Her father
was a Swedish sailor with a disabling arthritis, who died at age 72 of unknown cause. Her mother
developed kidney disease in her 40s and died edematous at age 68 of either renal failure or
congestive heart failure. There were three brothers, one who died in his 80s of metastatic bone
cancer. The medical histories of the other two are unknown. The patient was evaluated medically
just once, when she was 26, at the Boston City Hospital. After a week of observation and tests
failed to produce a diagnosis, she was told “to just go on living as [she] had always done. When the
patient was 56, she developed a severe illness thought to have been pneumonia. One evening,
while recuperating, she sat with one leg stretched out beneath a stove and fell asleep. When she
awoke, the heat from the fire had seared the flesh from her withered leg. The third-degree burn
healed slowly in response to repeated application of cod liver oil. At age 74, the patient finally
consented to the use of a wheelchair and died shortly thereafter.
Case: “Patient as Art” – Christina’s
World
No mental health issues
Family history: Arthritis , Kidney Disease, Bone Cancer
Swedish/Scandinavian
Blonde Hair
Odd Gait
Misshaped hands
Fell Frequently
Crab-like Gait
Third-degree burn
Lost ability to stand
Pneumonia
Age
Birth
3
13
19
26
50
56
74
Death
Relevance Mapping and Disambiguation
Case Re-annotation
Patient ‘Is a’ Female.
Patient ‘Has’ Blonde Hair
Patient ‘became’ Crippled
Patient ‘is’ Scandinavian
.
.
Disease ‘Affects’ Patient
Disease ‘Is a’ Degenerative Disorder
Disease ‘Is a’ Neuromuscular Disease
.
.
.
.
.
Disease ‘Affects’ Child
Disease ‘Affects’ Women
Disease ‘Affects’ Gait
Disease ‘Causes’ Standing Pain
Disease ‘Co-exists with’ Distal Muscle Weakness
Patient has normal childhood
Google Search
Step 1: Automatic Case-Context Generation
Step 2: Reasoning with N-ary associations
Neuromuscular disease - Distal muscle weakness
Gait - Distal muscle weakness
Gait abnormality - Falls frequently
Step 3: Hypothesis from Random Walks
Case vs. Control Random Walk
Disease
Symptom
Hypothesis / Results – Charcot Marie
Tooth Not Polio as in Art/History Books
Hypothesis
Probability
Hereditary Motor and Sensory Neuropathies
Category
Charcot Marie Tooth Disease
1
Welander Distal Myopathy
(common in Sweden)
2
Fasciitis Plantar
Talocalcaneal coalition
Cerebellar atrophy
Friedreich Ataxia
Hypolipoproteinemia
Multi infarct state
Neuroleptic Induced Parkinson
Quadriplegic spastic cerebral palsy
Subcortical vascular encephalopathy
Base probability for random disease : 1e-6
Mining for potential causes and credibility
Evaluating relevance to weak observations
Blonde_hair
CAUSES Tyrosinase_related COEXISTS
(Rev)
_protein_1
WITH
SCANDINAVIAN
PART
OF (Rev)
9p21
PART_OF
PER2_protei COEXIST
n__mammali S WITH
an
(Rev)
11q22
CAUSES
MTMR2
ASSOCIATED Charcot_Marie_Toot
WITH
h_Disease
Charcot_Marie_Tooth_Disease
Opening up previous year cases…
Character
Diagnosis (Expert)
ORiGAMI
Hypothesis
2014 – Oliver Cromwell
Malaria, Typhoid, Salmonella
infection
Malaria, Sepsis, Urinary Tract
Infection
2006 – Booker T. Washington
Nephrosclerosis, Hypertensive
cardiomyopathy
Acute congestive heart failure,
atrial flutter
2004 – Schliemann
Temporal lobe abscess/
exostoses of the external
auditory canal, Post-operative
meningitis
Actinomycotic brain abscess,
Enterovirus Infection
2002 – Herod
Uremia complicated by Fournier
gangrene, Generalized
atherosclerosis/hypertension
Fournier Gangrene, Deep Vein
Thrombosis, Pulmonary Edema
2001 - Claudius
Congenital dystonia/Amanita
mushroom poisoning,
Atherosclerosis
Sclerosing lipogranuloma, Acute
toxic hepatitis
2000 – Mozart
Acute rheumatic fever, poststreptococcus equiglomerulonephritis
Fever disorder, Legionnaires
disease
2003 – Florence Nightingale
Bipolar disorder, PTSD, (Heart
failure/old age)
Vascular dementia, atrial flutter,
high pressure neurological
syndrome
Top results listed….
Lessons Learned
• Positive Bias in Literature
– Not enough negations (elimination) in literature
• Fusion of Data (Statistics) and Meta-Data (Text)
– Quality of publication, size of control group, etc.
• Disambiguation
terminology
and
resolution
with
hierarchies
in
– Can be handled with advanced computing architectures
• Mapping of Spoken-English to Medical-Speak
– Solvable in the near future with “deep learning” techniques that
translate languages today.
Major Advances in Non-Medical
Artificial Intelligence
Nature February 2015
Groundbreaking Article
Closest So Far to AI for Radiology?
150
The Atlantic March 28, 2016
“Go” Was Called the “Holy Grail” of AI
Not Achievable for Another Decade
How Google’s AlphaGo Beat Lee Sedol
• Most major South Korean television networks are
carrying the game. In China, 60 million people are
tuning in
• A few hundred members of the press are in adjacent
rooms, watching the game alongside expert
commentators
• Potential board positions:
• 208,168,199,381,979,984,699,478,633,344,862,770,2
86,522,453,884,530,548,425,639,456,820,927,419,61
2,738,015,378,525,648,451,698,519,643,907,259,916,
015,628,128,546,089,888,314,427,129,715,319,317,5
57,736,620,397,247,064,840,935 more than atoms in
the Universe
Go
• Before the match, Lee claimed that the challenge wasn’t
whether he would beat AlphaGo, but whether it would be 5-0
or 4-1
• Other Korean players stated that it was the easiest million
dollars a top level player could make
• Lee goes on to lose Game 1, resigning after 186 moves
• In game 2 AlphaGo plays a move 37 after which Lee walks
out of the room, he resigns after 211 moves
• After losing game 3, Lee apologizes to the entire world, “I
apologize for being able to satisfy a lot of people’s
expectations”
• Lee went on to win game 4 with a “hand of God” move at
turn 78 and then lose game 5 using the same strategy
Does Artificial Intelligence
Just Emerge With Enough
Speed and Memory?
•
When Mike was installed in Luna, he was pure thinkum, a flexible
•
logic--"High-Optional, Logical, Multi-Evaluating Supervisor, Mark IV,
•
Mod. L"--a HOLMES FOUR. He computed ballistics for pilotless freighters
•
and controlled their catapult. This kept him busy less than one percent
•
of time and Luna Authority never believed in idle hands. They kept
•
hooking hardware into him--decision-action boxes to let him boss other
•
computers, bank on bank of additional memories, more banks of
•
associational neural nets, another tubful of twelve-digit random numbers,
•
a greatly augmented temporary memory. Human brain has around ten-to-the tenth neurons. By third year
Mike had better than one and a half times that number of neuristors.
•
And woke up.
•
He winked lights at me. "Hello, Man."
•
"What do you know?"
•
He hesitated. I know--machines don't hesitate. But remember, Mike was designed to operate on incomplete
data. Lately he had reprogrammed himself to put emphasis on words; his hesitations were dramatic. Maybe
he spent pauses stirring random numbers to see how they matched his
•
Memories.
•
"'In the beginning,'" Mike intoned, "God created the heaven and the earth. And the earth was without form,
and void; and darkness was upon
•
the face of the deep. And--'"
•
"Hold it!" I said. "Cancel. Run everything back to zero.”
Can We Really Trust Deep
Learning Algorithms to Drive or
Practice Medicine? How Do We
Debug Them?
But Could AI Ever Be
Creative?
If equal to human which
human?
Prehistoric Man
Me?
Tanveer, Khan, and
Rasu?
Wired Magazine
Conclusion
Reasons Why Radiologists Won’t be
Replaced Any Time Soon
• There are tens of thousands of algorithms that
have been developed for image analysis and
decision support over the past 30 years and for
the most part none are in clinical practice
• In order to replace a radiologist, someone would
have to find the best of these and consolidate
them into a package that could work
independently (unsupervised) for image review
but these are written in different “languages” with
different assumptions about the images
• Each of these algorithms is generally super
narrow, so in order to replace a radiologist, one
would have to have a general portfolio that did
everything all specialists do currently
• The work on computer vision recognizing water bottles
in an image database is fundamentally different from
diagnostic images including the fact that the images are
24 bit color (8 bit) and that there is no algorithm or
methodology that is comparable for these image
challenges for diagnostic radiology
• Assuming you had all of these
available and somehow integrated
you would then have to start
getting FDA approval which could
take another 30 years for each
and every one given resources
and rate of approval of software
• Let’s assume that you actually
discovered/created a program so “smart”
that it could read textbooks and journal
articles and review prior images on PACS
and reports in the EMR and that it actually
was better than any subspecialty radiologist
at all tasks (far fetched from today’s reality)
• If so, then how would you test it to make
sure it had knowledge in all of those areas
satisfactorily
Will Computers Replace Radiologists
Soon?
• How long would it take to complete those tests? Years?
•
•
•
•
•
Decades? We barely know how to test humans in these
areas.
How long would it take to get the datasets ready to train
these
How long would it take to get FDA approval for these
What would be the medico-legal issues associated with
implementation of these?
How long would it take people to purchase these and
what would they cost given the cost of development?
Much more likely incremental changes that will be
informed by machine learning
Incredibly Exciting Potential for Machine
Learning in Medicine and Diagnostic
Imaging
• Intelligent screening criteria for mammography, lung cancer, and other
•
•
•
•
•
•
•
•
•
•
cancers including genomic/liquid biopsy data and other lab info
Patients at risk for contrast
Automatic protocoling of studies
Smart PACS hanging protocols and synchronization protocols
Smart transfer of findings from workstation to speech recognition
Assessment of patients at high risk to have positive findings (or low
risk)
Communication and tracking of findings
Multiparametric analysis across multiple modalities
Improved departmental efficiency with decreased waiting times
Dose optimization
Quality improvement in scanning
• So I’m here to tell all of the “worthless
protoplasm” that are radiologists and
the rest of you human beings that you
can continue to ingest food,
reproduce, and create waste products
without fear of being replaced by
computers, at least in radiology, any
time soon!
170
And Finally, It’s Not Easy To Tweet Like
a Human: Microsoft Bot
171
172
173
Conclusions:
So João
• Finish your residency
• Lots of work to do before we have a general
ML, DL, AI program that can learn radiology
conceptually, rapidly, like radiology resident
• Loads of low hanging fruit for machine
learning techniques that we should be
pursuing much more aggressively now
Humans Have Been Around 1 minute
17 seconds - Computers closer to a
Millisecond. Who Knows What Will
Happen in the Next Microsecond?!
Summer Reading?
176
Coming Events
• SIIM AI Conference and potential
role of SIIM with Dr. Brad Erickson
• Debate at RSNA about whether
radiologists will be replaced in 20
years also Dr. Bradley Erickson
Eliot Siegel, MD, FSIIM, FACR
Professor of Radiology
University of Maryland School of Medicine
Chief Imaging Services, VA Maryland Healthcare System