Download slides

ESS011
Mathematical statistics and signal processing
Lecture 5: Independence of events and rules involving conditional
probabilities
Tuomas A. Rajala
Chalmers TU
March 24, 2014
Course ESS011 (2014)
Lecture 5: Independence of events and rules involving conditional probabilities
Where are we
Last week:
Probability is a quantification of uncertainty: Random experiments
happen. A lot.
Relative frequency vs classical probability: Experience vs. logic. Can
we repeat the experiment?
We translate our questions into events in sample spaces
Probability is the measure of the events w.r.t. the sample space
and
How to count: multiplication rule, combinations, permutations
The three axioms of probability theory
We continue today with two important concepts: Conditional
probabilities and the independence of events.
1/18
Course ESS011 (2014)
Lecture 5: Independence of events and rules involving conditional probabilities
Motivation
Example: A family has two children. What is the probability that the
younger child is a girl given that at least one of them is a girl?
Remember the and word: P (A and B) = P (A ∩ B)
We also know the or word: P (A or B) = P (A ∪ B).
These are not enough to answer the question: we need additionally the
”probability of B if A holds”.
2/18
Course ESS011 (2014)
Lecture 5: Independence of events and rules involving conditional probabilities
Conditional probability
Conditional probability For events A and B with P (B) > 0, the
conditional probability of A given B is defined as
P (A|B) :=
P (A ∩ B)
P (B)
Terminology: ”If B holds, what is prob. of A”, ”What is prob. of A
given B holds”
Girl example: The sample space is S = {bb, bg, gb, gg}. We want
Ayg = {younger is a girl} = {bg, gg}.
Condition is Ag = {bg, gb, gg}.
The answer is P (Ayg |Ag ) =
P (Ayg ∩ Ag )
2/4
2
=
= .
P (Ag )
3/4
3
3/18
Course ESS011 (2014)
Lecture 5: Independence of events and rules involving conditional probabilities
Conditional probability, another example
Example: You play a game with your friend. Your friend rolls 2 dice,
hidden from you. He says ”the sum is above 5 and below 10”. Now, if
you guess the numbers you win a price. Wrong answer lowers the price.
You consider saying ”at least one number 3”. What are the chances?
Write A3 := {at least one 3}, As := {sum is above 5, below 10}.
Task: P (A3 |As ).
We can compute
P (A3 ∩ As ) =
7
,
36
P (As ) =
20
36
then by definition,
P (A3 |As ) =
As P (A3 ) =
7
≈ 0.35
20
11
≈ 0.31, knowing As changes the chances of A3 .
36
4/18
Course ESS011 (2014)
Lecture 5: Independence of events and rules involving conditional probabilities
Independence of events
Independent objects = not connected with each other, not influencing
each other
Example: Roll 2 dice, let B := {first is 1 or 4} and C := {second is 3}.
Now
2
1
12
=
and P (C) =
P (B) =
36
6
6
Furthermore
P (B ∩ C) = P ({(1, 3), (4, 3)}) =
2
= P (B)P (C).
36
But for the child example
P (Ayg )P (Ag ) =
2
2 3
· =
6
= P (Aag ∩ Ag )
4 4
4
Key: Events B and C do not contain information about each other.
5/18
Course ESS011 (2014)
Lecture 5: Independence of events and rules involving conditional probabilities
Independence of events
Independent events Events A and B are independent if and only if
(”iff”)
P (A ∩ B) = P (A)P (B)
We denote independence of A and B by A ⊥
⊥ B.
Furthermore, let C = {A1 , A2 , ..., An } be a collection of events:
Independent collection Events in C are independent iff for any
subcollection A(1) , A(2) , ..., A(m) such that (i) 6= (j)
P (A(1) ∩ A(2) ... ∩ A(m) )
=
=
P (A(1) )P (A(2) ) · · · P (A(m) )
m
Y
P (A(i) )
i=1
(super useful for statistics)
6/18
Course ESS011 (2014)
Lecture 5: Independence of events and rules involving conditional probabilities
Independence of events: Example
Example: Flip a fair coin, then another. As the flips are independent, the
probability for two heads
P (H ∩ H) = P (H)P (H) =
1
11
=
22
4
Repeat p times, get P (H)p = 1/2p .
Another: You’re playing red on roulette. You have lost two times. Is the
next round your turn to win? Roulette rounds are independent, so
P (red|twice not red) =
P (red)P (twice not red)
P (red ∩ {twice not red})
=
P (twice not red)
P (twice not red)
= P (red). The past does not matter in roulette.
7/18
Course ESS011 (2014)
Lecture 5: Independence of events and rules involving conditional probabilities
Multiplication rule
We now know how to derive P (A ∩ B) if A ⊥
⊥ B: it is P (A)P (B). What
if they are not independent?
Rearrange the conditional probability formula:
Multiplication rule For events A and B,
P (A ∩ B) = P (A|B)P (B)
The multiplication rule is can be generalized to
the chain rule of probability: For events A, B, C
P (A ∩ B ∩ C) = P (A|B ∩ C)P (B|C)P (C)
and so forth for more events.
8/18
Course ESS011 (2014)
Lecture 5: Independence of events and rules involving conditional probabilities
Example of multiplication rule
Example: Draw 2 cards from a well shuffled ordinary deck of cards. What
is the probability that a) You get two aces? b) The second card is ace?
Write Ai = {ith card ace}, i = 1, 2.
a) We want P (A1 ∩ A2 ). As P (A1 ) = 4/52 and P (A2 |A1 ) = 3/51, we have
P (A1 ∩ A2 ) = P (A2 |A1 )P (A1 ) =
3·4
≈ .005
51 · 52
b) Two cases can occur: A1 ∩ A2 or A01 ∩ A2 . They are mutually exclusive, so
P (A2 ) = P ((A1 ∩ A2 ) ∪ (A01 ∩ A2 )) = P (A1 ∩ A2 ) + P (A01 ∩ A2 )
and similarly as in a) P (A01 ∩ A2 ) =
4 · 48
3 · 4 + 4 · 48
, so P (A2 ) =
≈ .08
51 · 52
51 · 52
***
9/18
Course ESS011 (2014)
Lecture 5: Independence of events and rules involving conditional probabilities
Law of total probability
Partition A collection of n mutually exclusive events that cover the
sample space,
C = {Bi : Bi ∩ Bj = ∅ for all i 6= j,
n
[
Bi = S}
i=1
is called the partition of S.
Note:
For any A, {A, A0 } is a partition.
The events in a partition can not happen at the same time.
At least one of the events in partition must have P (Ai ) > 0.
10/18
Course ESS011 (2014)
Lecture 5: Independence of events and rules involving conditional probabilities
Law of total probability
Using the multiplication rule, we get
Law of total probability For a partition A1 , A2 , ..., An
P (B) =
n
X
P (B|Ai )P (Ai )
i=1
Proof: Note that B = B ∩ S, and use distributive law of sets.
The law provides a decomposition of the probability into conditional
events.
Example: In the ”draw 2, second is ace”, we knew P (A2 |A1 ) and
P (A2 |A01 ), and so law of total probability gives directly
P (A2 ) = P (A2 |A1 )P (A1 ) + P (A2 |A01 )P (A01 )
.
11/18
Course ESS011 (2014)
Lecture 5: Independence of events and rules involving conditional probabilities
Bayes’s theorem
The last and the most useful law for applications in modern statistics:
Assume that the coin is fair, P (H) = 1 − P (T ) = 0.5. Then you can
derive other events. This is mathematics.
But what if all you know is that you got 6 heads out of 10 tosses: Is the
coin fair? After all, the outcome is random, P (H) could be 0.5 or not.
This is statistical question, sometimes called the inversion, and is solved
later. The solution rest on the following:
Bayes’s theorem Let A and B be events such that P (B) > 0. Then
P (A|B) =
P (B|A)P (A)
P (B)
Derivation: Use conditional probability twice.
(Reverend Thomas Bayes, posthumously published works 1764)
12/18
Course ESS011 (2014)
Lecture 5: Independence of events and rules involving conditional probabilities
Bayes’s theorem: general case
In Bayes’s theorem the denominator can be written as
P (B) = P (B|A)P (A) + P (B|A0 )P (A0 )
Often we have more than one event we need to consider, i.e. a partition.
By law of total probability:
The general form of Bayes’s theorem Let A1 , ..., An be a partition of
S. Let B be an event such that P (B) > 0. Then
P (B|Ai )P (Ai )
P (Ai |B) = Pn
j=1 P (B|Aj )P (Aj )
13/18
Course ESS011 (2014)
Lecture 5: Independence of events and rules involving conditional probabilities
Bayes’s theorem: Classical example
AIDS testing. A blood test for AIDS checks for specific antibody markers
in the blood sample. It has been found that the test is 99.9% succesful in
capturing AIDS cases (”true positive rate”).
You have just been to a routine checkup. The doctor calls in the news:
”Your test for AIDS was unfortunately positive... you, sir, have AIDS.
The margin of error is only 0.1%”.
Hmm... you know this is highly unlikely. Write A = {have AIDS},
T = {test is positive}. What the doctor now (wrongly) reports to you is
P (T |A). Is this what we want to know?
14/18
Course ESS011 (2014)
Lecture 5: Independence of events and rules involving conditional probabilities
Bayes’s theorem: Example contd.
What we really want to know is P (A|T ): Do we have AIDS if the test is
positive. Enter Bayes:
P (A|T ) =
P (T |A)P (A)
P (T )
We need one extra data: The overall prevalence of AIDS, P (A). It is
approx. 0.2% in Swedish adult population.
Then, using the general form of Bayes’s formula:
P (A|T ) =
0.999 · 0.002
≈ 0.67
0.999 · 0.002 + 0.001 · 0.998
so there is still a good change that we are fine: roughly 33 out of every
100 tested will be diagnosed wrongly as ill (”false positive”).
15/18
Course ESS011 (2014)
Lecture 5: Independence of events and rules involving conditional probabilities
Recap of basic probability theory 1/2
The basic of probability theory: For a random experiment with a sample
space S and events A, B ⊆ S,
The axioms: P is a probability if P (S) = 1, P (A) ≥ 0 and
P (A ∪ B) = P (A) + P (B) whenever A ∩ B = ∅
The general addition rule:
P (A ∪ B) = P (A) + P (B) − P (A ∩ B)
Independence of events:
A⊥
⊥ B ⇔ P (A ∩ B) = P (A)P (B)
16/18
Course ESS011 (2014)
Lecture 5: Independence of events and rules involving conditional probabilities
Recap of basic probability theory 2/2
Conditional probability: When P (B) > 0,
P (A|B) =
P (A ∩ B)
P (B)
Multiplication rule:
P (A ∩ B) = P (B|A)P (A)
Law of total probability: For a partition A1 , ..., An of S
P (B) =
n
X
P (B|Ai )P (Ai )
i=1
Bayes’s theorem: when P (B) > 0,
P (B|A) =
P (A|B)P (B)
P (B)
17/18
Course ESS011 (2014)
Lecture 5: Independence of events and rules involving conditional probabilities
Summary of today’s lecture
Now we know
How to describe ”randomness” and ”uncertainty” with probabilities
of a random experiment
How to derive unknown probabilities from known probabilities
The probabilities so far have come from
a) Logical computations based on counts of possible outcomes
b) Numbers estimated by others; experience
In the next lecture we will go into stochasic models for random events,
and start to approach the connection between probability theory and
reality.
18/18