Download Probability

Probability Intro
Professor Jim Ritcey
EE 416
Revised Fall 2013
Please elaborate with your own sketches
Disclaimer
• These notes are not complete, but they should
help in organizing the class flow.
• Please augment these notes with your own
sketches and math. You need to actively
participate.
• It is virtually impossible to learn this from a
verbal description or these ppt bullet points. You
must create your own illustrations and actively
solve problems.
Random Experiments
• An experiment is prescribed with N outcomes
• finite {1 2 3} countable {1 2 3 …} uncountable {z>0}
• A chance mechanism selects outcome – uncertainty
• We have some knowledge of the likelihood of
occurrence – often measured empirically through
relative frequency in repeated independent trials
• Kolmogorov developed consistent axiomatic
framework for probability
Relationship to Set Theory
• Set Theory
Probability
• Universe
• Elements
• Algebra of subsets
Sample space
Outcomes
Events
• To each event the theory allows/assigns a number
• Probability (event) indicates its likelihood of
occurrence 0<= p <= 1
Set Operations – PPT Notation
• Complement
A^c = {c: c in S but c not in A}
• Union
(cup) A+B = {c: c in A or c in B }
• Intersection (cap) AB = {c: c in A and c in B }
• Often we use cap/cup/overbar (complement) in
traditional math notation
• MATLAB provides set operations as functions very useful when working with large sets
Algebra 0f Events
• Each experiment has outcomes in sample space S
• Subsets of outcomes are called events.
• The set of events must be an algebra –closure
•
•
•
•
For events A,B,C, … then
All complements are events A^c, B^c, …
All finite unions & intersections are events
All countable unions & intersections are events
• For finite sample spaces the events consist of the
set of all subsets. But too large for infinite S
Probability Triple
• (S, E, P) (Sample Space, Events, Probability)
• Sample space – set of all possible outcomes
• Events – family/collection of all subsets to which
we assign probabilities. Subject to closure under
set operations
• Probability – maps events to unit interval [0,1]
subject to 3 Axioms. For events A,B,C,…
• (1) P(A) >= 0 non-negative probability
• (2) P(S) = 1 something happens w probability 1
• (3) A, B disjoint P(A+B) = P(A) + P(B)
• (3’) Can extend (3) to countable unions
Examples
•
•
•
•
S = {T,H} = {0,1} 1 coin toss
E = { S^c = {}, {0}, {1}, S = {0,1} }
Given P(1) can compute probability of any e in E
In the model, p=P(1), is a parameter
• S = {TT,HT,TH,HH} = {00,10,01,11} 2 coin toss
• E = { S^c = {}, {0}, {1}, S = {0,1} }
• Given P(1) can compute probability of any e in E
• Each event can be given an intuitive name
S = {00,10,01,11}
{H=1,T=0}
e
0
1
2
3
outcomes
1
0
0
0
0
Empty set = nothing happens
2
0
0
0
1
Exactly {11} “2 heads”
3
0
0
1
0
Exactly {01} “tail then head”
4
0
0
1
1
{01} or {11} “head on second toss}
5
0
1
0
0
Exactly {10}
1
0
0
1
Exactly “2 of the same”
1
1
1
0
“Anything but 2 heads” complement of #2
6
7
10
14
15
16
S = something happens
Probability of an Event
• Event is a subset of outcomes
• An event is realized (occurs) if any of its members
occurs
• The outcomes of an experiment are by definition
disjoint
• P (event) is the sum of the probabilities of
outcomes that define it 
• P(event) = P(outcome_1) + …
• But events can contain many outcomes
How to we know P
• P( any event ) is known in the theory. But how?
• Relative frequency in independent repeated trials
• Uniformity considerations – insufficient reason to
suspect otherwise “at random”
• Subjective – expert opinion, must be consistent
• What is the probability that the sun will rise
tomorrow p=1? p<1?
• What is the probability that you will obtain above
3.4 in EE416?
• Engineering usage focuses on relative frequency
• Based on past ( but relevant) history
Theorems are derived from Axioms
• P(A^c) = 1 - P(A)
• P(S) = 1  P(empty set) = 0
• P(A+B) = P(A) + P(B) - P(AB)
• P(A+B) < = P(A) + P(B)
• A subset of B then P(A) <= P(B)
• Proofs are instructive usually they relate to
disjoint events
Partitions
•
•
•
•
If A us an event, and A in B in S (sample space)
Let B_1, …B_N partition B
Then
P(A) = P(AB_1) + … +P(AB_N)
• Simplifies more complicated events
General Problem of Computing Prob
• Given a combination of events, with knowing prob
of each separately, determine prob of combination
• Typical combinations “at least one” “k or more of”
• Always can be written as set operations
• Always can be enumerated, but can be a long list
1 Coin Toss
•
•
•
•
•
•
•
•
Experiment – toss a coin S –{T,H} |S| =2
Here let the events be the power set Pwr(S)
Set of all subsets with 2^2 -4 elements
S^c = Empty set, H, T, {H or T} =S closed family
Assign P(H) = p, p = 1/2 only for a fair coin
Consistent with axioms if p in [0,1] & P(T) = 1-p
Often insufficient reason to take p neq 1/2
Often p should be measured empirically, the
fraction of heads in many trials (how many?)
• The outcomes have prob {pT= 1-p, pH = p}
Geometric Probability
• In many board games spinners are used. You cut a
circular region into n congruent pizza slices, and
flip a spinner. It is obvious that the probability of
any outcome is p = 1/n.
• This can be argued to be the ratio of the arc
length of the wedge to arc length of the circle
• Geometric probabilities reduce to calculation of
relative length/area/volume and are based on an
“at random” assumption. This implies uniformity.
2 Coin Toss
• S = {TT, TH, HT, HH} ordered (T1, T2)
• Pwr(S) has 2^4 = 16 possible events
• It is not possible to combine outcomes in any
other way. Next assign a consistent probability to
• (p1, p2,p3,p4) p_i in [0,1] sum p_i = 1
• Insufficient reason takes p_i = ¼
• But it depends on the random experiment (toss)
• Allowable is any consistent set of probabilities.
• Then eg, P(T1=TorT2=T) = P(HT + TH + TT) =
=P(HT) + P(TH) + P(TT)
Independence
•
•
•
•
Two events A,B are independent when
P(AB) = P(A)P(B) multiply probabilities
This is a property of the probability assignment
Disjoint events are
• Dependent events – statistical relationship/linking
• This allows for prediction of one given the other
• Not a one-way causal relationship
2 Coin Toss
•
•
•
•
•
•
•
•
•
•
Label the outcomes (00, 01, 10, 11) where T=0,H=1
Suppose we measured
P(00) = 0.10 P(01) = 0.30
P(10) = 0.22 P(11) = 0.38
Are the tosses independent?
P(t1=0)=P(00)+P(01) = 0 .1+0.3 =0.40 P(t1=1)= 1-.4
P(t2=0)=P(00)+P(10) = 0.1+0.22=0.32 P(t2=1 = 1- .32
Independence says that
P(t1=0)P(t2=0) = P(00) but this does not hold!
Under this P, Events are not independent!
Are disjoint events independent? NO
• Let A,B mutually exclusive AB = empty set
• P(AB) = P(empty set) = 1-P(S) = 0
• They are not independent
• Disjoint events are highly dependent – if one event
occurs, the other cannot have occurred.
• This must be distinguished from independence
• P(AB)=P(A)P(B) which might be zero if P(A)=0
• A property of the Prob Assnment, not the events
Communications Networks
•
•
•
•
A network is made up of nodes and links (edges)
The link either work 1 or fail 0 with p=P(1) q=P(0)
Links can be connected in various ways
Find Probability P_N that a network works
• Series Net: all links work
P_N = p^N
• Parallel Net: at least 1 link works P_N = 1 - q^N
• Need to work through the derivation
• Networks can combine series and parallel
Target Detection
• A radar takes N looks at a target, target is
detected (when present) with p=0.9
• How does detection prob improve as we increase N
• P_N := 1 –(1-p)^N but we need to explore
numerically
• For small N,
• P = [ p, 2p-p^2, 3p -3p^2 +p^3, … ]
• Is this a series or parallel problem?
Birthday Problem
•
•
•
•
•
Birthdays occur at random on [1:365] no leap yr
Sample r people’s birthdays, n=365
find P(hit) =P( 2or more birthdays in sample)
P(no hit) = # ways to select wo rep/# ways w rep
1-P(hit)= (n)_r / n^r where (n)_r = n!/(n-r)!
• Numerical implications are interesting
Birthday Problem Numerics
• Place r balls at random in n bins. What is the probability that
2 or more land in any bin
• N=365, r=23 for a 50:50 chance of hits
• Randomness causes clumping!
• Table shows P(hit) vs r, number of balls placed
• Balls can be calls, bins can be channels/frequencies
20
0.4
23
0.507
30
0.706
35
0.814
40
0.89
Birthday Problem Analysis
•
•
•
•
•
•
•
Give a simple expression for
P(r,n) = 1 –(n)_r/n^r, when n is large
Write Q = 1-P as a product
Q=1-P(r,n) (n)_r/n^r = Prod_1^(r-1) [1-j/n]
Go to a log form exp( sum of logs)
Approximate the sum & simplify
This is computational probability
• Or just compute use Gamma functions
Birthday Prob (n=365)
Matlab Code
•
•
•
•
•
function birthdayprob( N, K);
%function birthdaypob( N, K);
%compute all birthday probabilities.
kall =[0:K-1];
%Pnohit =(N)_k/N^k;
•
•
•
•
•
•
Pnohit = exp( cumsum( log(1 -kall/N) ) );
Pnoapp = exp( -(kall+1).*(kall)/(2*N) );
% a very good approximation!
X = [1:K]'*[ 1 1];
P1 = [ Pnohit;1-Pnohit]';
P2 = [ Pnoapp; 1-Pnoapp]';
•
•
•
•
•
% note that we are plotting both tails
figure(1); semilogy( X, P1,'-*');
hold on; semilogy( X, P2,'-o'); hold off
title('Exact vs Approx hit Probabilities');
xlabel('Number of Balls Thrown');grid;
Ranking Problems
• You meet Alice, she has two brothers Bob and
Chris. You don’t know anything more.
• What is the probability that Alice is the oldest?
• Use Insufficient Reason to determine
• P(A oldest) = 1/3
• Now she tells you that Alice is older than Bob.
• Determine P(A is oldest).
• Hint: Consider the simplest sample space