Download Lecture1

Thomas Bayes
(1702-1761)
Pierre-Simon Laplace
(1749-1827)
Bayesian Reasoning
A/Prof Geraint Lewis
A/Prof Peter Tuthill
“Probability theory is nothing but common sense, reduced to calculation.”
Laplace
Are you a Bayesian or Frequentist?
4
“There are 3 kinds of lies: Lies, Damned Lies, and Statistics”
...and Bayesian Statistics
Benjamin Disraeli
Frequentists
Fig 1. A Frequentist Statistician
Fig 2. Bayesian Statistics Conference
What is Inference?
If A is true then B is true (Major Premise)
A = A,B
(in Boolean notation)
Deductive Inference (Logic)
Aristotle 4th Century B.C.
A is true (Minor Premise)
therefore B is true (conclusion)
B is False (Minor Premise)
therefore A is False (conclusion)
}
A B
T →T
STRONG
SYLLOGISMS F ← F
Inductive Inference (Plausible Reasoning)
B is true (Minor Premise)
therefore A is more plausible
A is false (Minor Premise)
therefore B is less plausible
}
t←T
WEAK
SYLLOGISMS F → f
What is Inference?
Deductive Logic:
Cause
Effects
or
outcomes
Inductive Logic:
Possible
Causes
Effects
or
observations
What is a Probability?
Frequentists
Bayesians
P(A) = long run relative frequency
of A occurring in identical repeats
of an observation
P(A|B) = Real number measure of
the plausibility of proposition A,
given (conditional upon) the
truth of proposition B
“A” is restricted to propositions
about random variables
“A” can be any logical proposition
All probabilities are conditional;
we must be explicit what our
assumptions B are (no such thing
as an absolute probability!)
Probability depends on our state of
Knowledge
Monte
Hall
A
B
?
C
Probability depends on our state of
Knowledge
7 Red
5 Blue
?
1st draw
5/12 Blue
7/12 Red
2nd draw
The Desiderata of Bayesian
Probability Theory
• Degrees of plausibility are represented by real
numbers (higher degree of belief represented
by a larger number)
• With extra evidence supporting a proposition,
the plausibility should increase monotonically
up to a limit (certainty).
• Consistency. Multiple ways to arrive at a
conclusion must all produce the same answer
(see book for additional details)
Logic and Probability
• In the certainty limit, where probabilities go to
zero (falsehood) or one (truth), then the sum and
product rules reduce to formal Boolean deductive
logic (strong syllogisms).
• Bayesian Probability is therefore an extension of
formal logic into intermediate states of
knowledge.
• Bayesian inference gives a measure of our state
of knowledge about nature, not a measure of
nature itself.
The two rules underlying
probability theory
SUM RULE:
P(A|B) + P(A|B) = 1
PRODUCT RULE:
P(A,B|C) = P(A|C) P(B|A,C)
= P(B|C) P(A|B,C)
Left
Handed
Blue,
Blue Eyes
Left
Brown Eyes
Right
Handed
All
Kangaroos
Bayes’ Theorem
Posterior
Bayes Theorem:
P(Hi|D,I) =
P(Hi|I) P(D|Hi I)
P(D|I)
Hi = proposition asserting truth of a hypothesis of interest
I = proposition representing prior information
D = proposition representing the data
P(D|Hi I) = Likelihood: probability of obtaining the data
given that the hypothesis is true
P(Hi|I) = Prior: probability of hypothesis before new data
P(D|I) = Normalization factor (prob all hypothesis i sum to 1)
Example: The Gambler’s
coin problem
P(H|D,I) =
P(H|I) P(D|H I)
P(D|I)
Normalization factor – Ignore this for now as only need relative merit
Prior – what do we know about the coin?
Assume H=pdf(head) is uniformly distributed 0-1
Likelihood – if we assume the data D gives R heads in N tosses:
P(D|H I)  HR (1-H)N-R
The full distribution, assuming independence of
throws, is the Binomial Distribution. We omit terms
not containing H, and use a proportionality.
Example: A fair coin?
Data
H
H
T
T
Example: A fair coin?
The effects of the Prior