Download Document

Bayesian Approaches
1
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
2
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
3
Retrospective
Bayesian Multimodal Perception by J. F. Fereira
Bayes' theorem - Bayes rule
Knowledge of past behavior and state
form prediction of current state
Non-Gaussian likelihood functions
Multimodal Sensing in human perception
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
4
Retrospective
distribution of object position unknown => flat
Noise in each modality is independent
bimodal posterior distribution = product of the unimodal distributions
Simplification: Probability distributions are Gaussian
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
5
1. Introduction to Pattern Recognition
Example:
“Sorting incoming Fish on a
conveyor according to species
using optical sensing”
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
6
1. Introduction to Pattern Recognition
Example: Fish Classifier
Selecting length feature
Selecting lightness feature
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
7
1. Introduction to Pattern Recognition
Example: Fish Classifier
Search
Best
performance
fortwo
thefeatures
optimal
but and
Selecting
complicated
tradeoff
classifier
– will
defining between
a simple
straight
not
performance
well
on boundary
the
withtraining
novel
line perform
as decision
patterns
set
and simplicity
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
8
1. Introduction to Pattern Recognition
decision
Pattern Recognition System
post-processing
classification
feature extraction
segmentation
Invariant Features
Translation
Rotation
Error Rate
Scale
Noise
Risk
Occlusion
Missing
Context
Features
Projective Distortion
Multiple Classifiers
Rate
Deformation
Feature Selection
sensing
input
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
9
1. Introduction to Pattern Recognition
start
Design Cycle
collect data
choose features
choose model
Prior Knowledge
Overfitting
train classifier
evaluate classifier
end
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
10
2. Continouos Features
State of nature 
Finite set of c states of nature (‘categories’) {1, … , c}
Prior P(j)
If the state of nature is finite:
Decision rule (for c =2):
Decide 1 if P(1) > P(2); otherwise decide 2
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
11
2. Continouos Features
Feature vector x :
x  d the feature space
x is (for d=1) a continuous
random variable x
Class(State)-conditional
probability density
function: p(x| j)
expresses the distribution
of x depending on the
state of nature
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
12
2. Continouos Features
Bayes formula
(Posterior)
Evidence
Bayes Decision rule (for c =2):
Decide 1 if P(1 | x) > P(2 | x) ; otherwise decide 2
Bayes Decision rule (expressed in terms of Priors):
Decide 1 if p(x|1)P(1) > p(x|2)P(2) ; otherwise decide 2
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
13
2. Continouos Features
Bayes Risk
Conditional Risk
We can minimize our expected loss by selecting
the action that minimizes the conditional risk.
Two-Category Classification
Decide 1 if (21-11)P(1 | x) > (12-22)P(2 | x) ; otherwise decide 2
This Bayes decision procedure provides the optimal performance
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
14
Discrete Features
Posterior
Feature vector x can assume m
discrete values
Evidence
Probabilities rather than probability
densities.
Risk
Bayes decision rule
To minimize the overall risk, select
the action I for which R(i|x) is
minimum
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
15
Discrete Features
Example: Independent Binary Features
2 category problem
Feature vector x = {x1, …, xd}T where xi = {0;1}
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
16
Bayesian Belief Networks
node
link
Represents knowledge about a
distribution.
variable
P(a)
parents
(of C)
children
(of E)
Knowledge:
Statistical Dependencies –
Causal Relations among the
component variables
Knowledge from e.g. structural
information
Graphical representation:
Bayesian Belief Nets
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
17
Bayesian Belief Networks
Applying Bayes rule to determine the probability of
any configuration of variables in the joint distribution.
P(a1)
P(a2)
0.739 0.261
Discrete Case:
Discrete number of possible values A
=1 (e.g. 2: a={a1, a2} and
continues-valued probabilities
P(c1|ak) P(c2|ak)
a1
0.3
0.7
=1
a2
0.6
0.4
=1
Conditional Probability Table
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
18
Bayesian Belief Networks
P(a)
A
P(b|a)
B
P(c|b)
C
Determining the
P(d|c)
probabilities of the variables
D E.g.: Probability distribution over d1, d2, … at D
Summing the full joint distribution
P(a,b,c,d) over all variables other than d
independance
simple split
simple interpretation
P(d)
P(c)
P(b)
Probability of a particular value of D
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
19
Bayesian Belief Networks
Give the values of some variables (evidence e)
… and search to determine some particular
configuration of other variables x
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
20
Bayesian Belief Networks
Example: Belief Network for Fish
Ex.2 Classify the fish:
Known: Fish is light (c1) and caught in the
south Atlantic (b2).
Unknown: Time of year (a), thickness (d)
As usual: Compute P(x1 salmon) and P(x2 sea
bass) Decide for the minimum expected
classification error
In this case D does not affect our results
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
21
Bayesian Belief Networks
Example: Belief Network for Fish
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
22
Bayesian Belief Networks
Example: Belief Network for Fish
After normalization:
And if the dependency relation is unknown?
naïve Bayes – idiot Bayes
Features are conditionally independant
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
23
Compound Bayesian Decision Theory
States of nature 
= ((1), … , (n))T
taking one of c values {1, … , c}
Consecutive ’s not statistically
independent
=> exploit dependence
=> improved performance
Prior P()
for n states of nature
Wait for n states to emerge and
make all n decisions jointly
= compound decision problem
Feature matrix X :
=(x1, …, xn)
xi obtained when state of nature
was i
n observations
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
24
Compound Bayesian Decision Theory
Posterior
Conditional probability
density function: p(X|)
for X given the true set of 
Define loss matrix for the compound
decision problem. Seek decision
rule the minimizes the compound
risk (optimal procedure)
Assumption: Correct = no loss
Errors = equally costly
=> simply calculate P(|X) for all 
and select  for which P(.) is
maximum.
joint density
practice: calculate P(|X) is time
expensive
assumption: xi depends only on (i)
not on other x or 
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
25
Obrigado!
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
26
Annex
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
27
Book: Pattern Cl.
Preface
Ch 1: Introduction
Ch 2: Bayesian Decision Theory
Ch 3: Maximum Likelihood and Bayesian Estimation
Ch 4: Nonparametric Techniques
Ch 5: Linear Discriminant Functions
Ch 6: Multilayer Neural Networks
Ch 7: Stochastic Methods
Ch 8: Nonmetric Methods
Ch 9: Algorithm-Independent Machine Learning
Ch 10: Unsupervised Learning and Clustering
App A: Mathematical Foundations
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
28
Book: Principles of ...
2
Bug Algorithms
17
3
Configuration Space
39
4
Potential Functions
77
5
Roadmaps
107
6
Cell Decompositions
161
7
Sampling-Based Algorithms
197
8
Kalman Filtering
269
9
Bayesian Methods
10
Robot Dynamics
349
11
Trajectory Planning
373
12
Nonholonomic and Underactuated Systems
401
301
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
29
Book: Artificial ...
Preface
Part I Artificial Intelligence
Part II Problem Solving
Part III Knowledge and Reasoning
Part IV Planning
Part V Uncertain Knowledge and
Reasoning
Part VI Learning
Part VII Communicating, Perceiving, and Acting
Part VIII Conclusions
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
30
Book: Bayesian ...
Preface xix
Part I: Fundamentals of Bayesian Inference 1
1 Background 3
2 Single-parameter models 33
3 Introduction to multiparameter models 73
4 Large-sample inference and frequency properties of Bayesian inference 101
Part II: Fundamentals of Bayesian Data Analysis 115
5 Hierarchical models 117
6 Model checking and improvement 157
7 Modeling accounting for data collection 197
8 Connections and challenges 247
9 General advice 259
Part III: Advanced Computation 273
10 Overview of computation 275
11 Posterior simulation 283
12 Approximations based on posterior modes 311
13 Special topics in computation 335
Part IV: Regression Models 351
14 Introduction to regression models 353
15 Hierarchical linear models 389
16 Generalized linear models 415
17 Models for robust inference 443
18 Mixture models 463
19 Multivariate models 481
20 Nonlinear models 497
21 Models for missing data 517
22 Decision analysis 541
Appendixes 571
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
31
Book: Classification ...
Preface.
Foreword.
1. Introduction.
2. Detection and Classification.
3. Parameter Estimation.
4. State Estimation.
5. Supervised Learning.
6. Feature Extraction and Selection.
7. Unsupervised Learning.
8. State Estimation in Practice.
9. Worked Out Examples.
Appendix
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
Images
32
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
33
2. Simple Example
Ten types of gestures:
1. Big circle
2. Small circle
3. Vertical Line
4. Horizontal Line
5. Pointing North-West
6. Pointing West
7. Talk louder
8. Talk more quiet
9. Wave Bye-Bye
10. I am hungry
Designing a simple classifier for gesture
recognition.
The observer tries to predict which
gesture might be performed next.
The sequence of gestures appears to be
random.
We assume that there is some a priori
probability (i.e. prior) P(1) that the next
gesture is ‘Big Circle’, P(2) that the
next gesture is ‘Small Circle’, etc.
If the gesture lexicon is finite:
State of nature  
Type of gesture (1 … 10)
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab
Bayesian Approaches
34
Missing and noisy features
Missing Features:
Example: x1 is missing
measured value of x2 is x^2
mean x1 points to omega 3
but omega2 better decision
Institute of Systems and Robotics
jrett
ISR – Coimbra
Mobile Robotics Lab