Download Statistical Independence

STATISTICAL INDEPENDENCE
AND
AN INVITATION TO THE Art OF CONDITIONING
Tutorial 2
STAT1301 Fall 2010
28SEP2010, MB103@HKU
By Joseph Dong
“Look, imagine a remote village where there has been a long
drought. One day, local peasants in desperation go to the
church, and the priest says a prayer for rain. And the next
day the rain arrives! These are independent events.”
Legendary answer made by Kolmogorov when he was questioned by a senior Soviet Minister
about whether the concept of independent events of his probability theory is consistent with
materialistic determinism.
What is the intuitive background of Independence? How is the concept of
independence related from that of mutual exclusiveness?
2
MOTIVATING CONCEPT INDEPENDENCE
•
The event of today’s raining is by no means affected by the prayer said at the
church yesterday.
•
How do you represent this wording in symbolic language so that you can analyze it in a
more precise manner?
•
Using the language of event (set):
• The realization of the event A={it rains today} is by no means affected by the
realization of event B={villagers prayed yesterday}.
• If B has already realized, A will have an unaffected probability of realizing.
• For convenience we use a notation Pr 𝐴|𝐵 to denote the probability of A’s
realization given B’s realization. Then the foregoing motivation for the concept
independence will require us to establish the precise meaning of Pr 𝐴|𝐵 = Pr 𝐴
in a symbolic way.
•
A and B independent ↭ Pr 𝐴 = Pr 𝐴|𝐵 = ??
3
INDEPENDENCE & MUTUAL EXCLUSIVENESS
•
Two events are mutually exclusive if there is no possibility that they can happen together
(simultaneously).
•
Mutual exclusiveness is a concept/phenomenon on the set theory layer.
•
Independence is a concept/phenomenon on the probability(measure) theory layer.
•
Two events are independent if … (see next slide)
4
Independence: Pr(𝐴𝑙𝑙 𝐸𝑣𝑒𝑛𝑡𝑠 ℎ𝑎𝑝𝑝𝑒𝑛) =
𝑃𝑟𝑜𝑑𝑢𝑐𝑡 𝑜𝑓 Pr 𝐸𝑎𝑐ℎ 𝐸𝑣𝑒𝑛𝑡 ℎ𝑎𝑝𝑝𝑒𝑛𝑠
•
A formal, less intuitive, precise, and deep definition: Two events are independent if
Pr(𝐵𝑜𝑡ℎ 𝐸𝑣𝑒𝑛𝑡𝑠 ℎ𝑎𝑝𝑝𝑒𝑛) = 𝑃𝑟𝑜𝑑𝑢𝑐𝑡 𝑜𝑓 Pr 𝐸𝑎𝑐ℎ 𝐸𝑣𝑒𝑛𝑡 ℎ𝑎𝑝𝑝𝑒𝑛𝑠
A = {The villagers prayed yesterday}, Pr 𝐴 =
3
,
10
B = {It rains today}, Pr 𝐵 =
1
3
Now we just need to specify one more probability to establish independence!
If events (A, B) are independent, so are events (A, Bc), (Ac, B), and (Ac, Bc).
Practice: handout problem 3.
5
NOW SHOW ALGEBRAICALLY…
•
If the events 𝐴, 𝐵 are independent, show that the pairs (𝐴, 𝐵 𝑐 ), (𝐴𝑐 , 𝐵), (𝐴𝑐 , 𝐵 𝑐 ) also
consist of independent events.
•
If the events 𝐴, 𝐵, and 𝐶 are completely independent (mutually independent), then the
pairs (𝐴, 𝐵𝐶), (𝐵, 𝐴𝐶), 𝐶, 𝐴𝐵 also consist of independent events.
•
𝑐𝑜𝑚𝑝𝑙𝑒𝑡𝑒 𝑖𝑛𝑑𝑒𝑝𝑒𝑛𝑑𝑒𝑛𝑐𝑒 ⇒ 𝑝𝑎𝑖𝑟𝑤𝑖𝑠𝑒 𝑖𝑛𝑑𝑒𝑝𝑒𝑛𝑑𝑒𝑛𝑐𝑒
•
P
ABC 
P
ABC  
c
 P
P
 A  P  B  P C 
 A  P  B  P C 
c


  P

AB 
 P
A P B 
But not the reverse:
• R.E. = Tossing a coin twice.
• A = {1st toss shows H}, B = {2 nd toss shows H}, C = {both tosses show the same side}
6
MORE ON
MUTUAL INDEPENDENCE
•
Intrinsic additive relations:
• the probabilities of the 4 vertices of a face add to the probability of the face.
• the probabilities of the 2 vertices of an edge add to the probability of the edge.
•
Mutual independence require us to establish all multiplicative relations among
vertices, edges, and faces.
7
MORE ON
MUTUAL INDEPENDENCE (CONT’D)
•
Pr 𝐴𝐵 𝑐 = Pr 𝐴 Pr 𝐵 𝑐
Pr 𝐴𝐵 = Pr 𝐴 Pr 𝐵 ⟹ Pr 𝐴𝑐 𝐵 = Pr 𝐴𝑐 Pr 𝐵
Pr 𝐴𝑐 𝐵 𝑐 = Pr 𝐴𝑐 Pr 𝐵 𝑐
•
Any two of the following three can derive the third.
• Pr 𝐴𝐵𝐶 = Pr 𝐴 Pr 𝐵 Pr 𝐶
• Pr 𝐴𝐵 = Pr 𝐴 Pr 𝐵
• Pr 𝐴𝐵𝐶 𝑐 = Pr 𝐴 Pr 𝐵 Pr (𝐶 𝑐 )
•
How to specify the multiplicative relations of the probabilities of a minimal set of
vertices, edges, and faces?
• Solution 1: 4 vertices
• Solution 2: ?
8
ANOTHER RELATED SOURCE OF PERSPECTIVE
Are the first two columns of digits
independent?
A
B
C
D
1
1
0
0
•
What about A and Bc? Ac and Bc?
0
1
1
0
•
What about second and third?
0
0
0
1
•
What about columns A & B together
and column C? If you are allowed to
add more rows, how could you make
them independent?
1
0
1
0
•
What about the first and last columns?
•
What about the first and last columns
together and column C?
•
9
A CONCRETE EXAMPLE
•
R.E. = Tossing a die once.
•
Event 𝐴 = 𝑜𝑑𝑑
•
Event 𝐵 = {𝑙𝑒𝑠𝑠 𝑡ℎ𝑎𝑛 𝑜𝑟 𝑒𝑞𝑢𝑎𝑙 𝑡𝑜 2}
•
What’s Pr 𝐴 and Pr 𝐵 ?
•
If you think A and B are not independent, then what can you say about Pr 𝐴𝐵 ?
•
Actually Pr 𝐴𝐵 =?
•
Are you surprised?
•
Can you give 3 more pairs of events that are independent?
10
CONDITIONAL PROBABILITY Pr ⋅ |𝐵
Pr 𝐴 𝐵
Pr 𝐴𝑐 𝐵
Pr 𝐴 𝐵𝑐
Pr 𝐴𝑐 𝐵𝑐
⋯⋯
Pr 𝐵𝑐 𝐴𝑐
=?
=?
=?
=?
=?
(Total: 4 × 2 = 8 conditional probabilities.)
Given an event 𝐵 (and of course also its sample space whereby it’s
contained), the probability conditional on event 𝐵, Pr ⋅ |𝐵 satisfies
Kolmogorov’s Axioms of Probability.
Pr(Ω|𝐵) =?
Pr 𝐴|𝐵 + Pr 𝐴𝑐 𝐵 =?
11
FORMAL DEFINITION: Pr ⋅ 𝐵 : =
Pr ⋅∩𝐵
Pr 𝐵
• Conditional Probability is a valid probability:
• Pr ⋅ 𝐵 ≥ 0?
• Pr Ω 𝐵 = 1? (“conditional sample space”)
• If 𝐴𝑖 ’s are disjoint events(subsets of Ω), then
Pr
𝐴𝑖 𝐵 =
Pr 𝐴𝑖 |𝐵 ?
• Exercise: Prove the identity
• Exercise: Prove the identity Pr 𝐴𝐵|𝐶 = Pr 𝐴|𝐵𝐶 Pr 𝐵|𝐶
• Exercise: Prove Pr 𝐴𝐵𝐶 = Pr 𝐴 Pr 𝐵|𝐴 Pr 𝐶|𝐴𝐵
• Practice: Handout Problems 1 and 2.
12
USING CONDITIONAL PROBABILITY ON ITS OWN
•
In English, we use “if”, “when”, and the alike to indicate a condition.
•
In Probabilistic analysis, we use the vertical bar “|” to indicate a condition.
•
If she says yes, we’ll adopt plan A; if she says no or is indifferent, we’ll go with plan B.
•
If the coin turns a head, I’ll go basketball tonight; if tail, 50% chance I’ll read a book
chapter on Bayes Theory, and 50% chance I’ll go to bed early.
•
If the airplane crashes in the prairie, then there is a high probability to find it; if in the
mountain, then medium probability to find it; if in the sea, then very low probability to find
it.
•
If tomorrow the announcement of last season’s financial result is above public
expectation, then the stock price will go up with probability 0.89; if equals the expectation,
the stock price will go up with probability 0.5; if below expectation, the stock price will go
up with probability 0.08.
13
PARTITION (OF SAMPLE SPACE)
AND CONDITION (ON THE PARTITION)
•
Formal Def. of a Partition of a Set.
• Any disjoint and exhaustive collection of subsets of a given set forms a
partition of that set.
•
A trivial partition of Ω: For each subset 𝐴 ⊂ Ω, 𝐴, 𝐴𝑐 is a partition of Ω.
•
Throwing a die 10 times and we are interested in the sequence of the ten outcomes. Then
Ω = {1111111111,
1111111112,
⋯,
6666666666}
If 𝐴𝑘 = 𝑡ℎ𝑜𝑠𝑒 𝑠𝑒𝑞𝑢𝑒𝑛𝑐𝑒𝑠 𝑤𝑖𝑡ℎ 𝑓𝑖𝑟𝑠𝑡 𝑜𝑢𝑡𝑐𝑜𝑚𝑒 𝑎𝑠 𝑘 , 𝑘 = 1, 2, ⋯ , 6 , then
Each 𝐴𝑘 is a subset of Ω
The six 𝐴𝑘 s are disjoint
The six 𝐴𝑘 s together are exhaustive.
The six 𝐴𝑘 s together is a partition of Ω.
14
EXERCISE
• We work in the sample space Ω. Let the collection of events
𝐴1 , 𝐴2 , ⋯ , 𝐴𝑛 be a partition of Ω. Also we have another event 𝐵 ⊂ Ω.
Prove the following identity:
• Further, prove the following additional identity:
15
PARTITION AND CONDITION
REVISITED – LAYMAN EDITION
•
You have made your first choice at box A, hoping that it contains the key to unlock a brand new
BMW. Then the host opens box B and it’s empty. Now it is your decision to make on whether
you will switch to choose C or stay with A.
•
What particular partition to form, and hence what conditions to make?
•
In a probability veteran’s language, it is said that we would “condition on A”
•
Meaning we first identify the event A as 𝐴 = 𝑡ℎ𝑒 𝑘𝑒𝑦 𝑖𝑠 𝑖𝑛 𝐴 .
•
Then form the trivial partition 𝐴, 𝐴𝑐 of the sample space.
•
Then you proceed the analysis by making a condition “if event X happens, then” for each
event X in the partition.
•
Here…
• if event 𝐴 happens (ie, …), and with the knowledge 𝐵 is empty, then 𝐶 must also be
empty.
• if event 𝐴𝑐 happens (ie, …), and with the knowledge 𝐵 is empty, then 𝐶 must contain
the key.
•
Now it is your turn to reach the end of this thread …
16
QUESTION TO THE LAYMAN EDITION
• What subtlety will be involved if, unfortunately,
you “conditioned on C”, instead of A?!
17
PARTITION AND CONDITION
REVISITED – EXPERT EDITION
•
•
Brief setup: Let 𝑋 = 𝑘𝑒𝑦 𝑖𝑠 𝑖𝑛 𝑏𝑜𝑥 𝑋 ,
where 𝑋 ∈ 𝐴, 𝐵, 𝐶
Recall an observation made in tutorial 1: What
are the two events involved when the host,
seeing you have chosen box A, shows you the
emptiness of box B?
•
Want: 𝐏𝐫 𝑨|𝛽, 𝑩𝒄 =? ?
•
Pr 𝛽, 𝐵 𝑐 |𝐴𝑐 = Pr 𝛽|𝐵 𝑐 , 𝐴𝑐 Pr 𝐵 𝑐 |𝐴𝑐 =
1
1×2
•
Pr 𝛽, 𝐵 𝑐 |𝐴 = Pr 𝛽|𝐵 𝑐 , 𝐴 Pr 𝐵 𝑐 |𝐴 = 2
Want =
1
•
Let 𝛽 = {ℎ𝑜𝑠𝑡 𝑐ℎ𝑜𝑜𝑠𝑒𝑠 𝐵}
•
•
Known:
Pr 𝛽, 𝐵 𝑐 |𝐴 𝑃(𝐴)
Pr 𝛽, 𝐵 𝑐 |𝐴𝑐 Pr 𝐴𝑐 + Pr 𝛽, 𝐵 𝑐 |𝐴 Pr 𝐴
1 1
×3
1
2
=
=
1 2 1 1 3
× + ×
2 3 2 3
Now it’s your turn: mimic an expert solution for the
“Three Prisoners Problem”.
1
2
•
Pr 𝛽|𝐴 =
•
Pr 𝛽|𝐴𝑐 , 𝐵 = 0
•
Pr 𝛽|𝐴𝑐 , 𝐵 𝑐 = 1
•
Pr 𝐴 = Pr 𝐵 = Pr 𝐶 =
1
3
18
EXERCISE
Handout Problem 4
•
What events are involved in the
statement:
• “The initial search was in the
mountains, and the plane was not
found.”
Handout Problem 5
•
Require a clever conditioning
(although the hint has effectively
eliminated this requirement) and some
mathematical maturity
19