Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Probability Intro Professor Jim Ritcey EE 416 Revised Oct 6 2009 Please elaborate with your own sketches Disclaimer • These notes are not complete, but they should help in organizing the class flow. • Please augment these notes with your own sketches and math. You need to actively participate. • It is virtually impossible to learn this from a verbal description or these ppt bullet points. You must create your own illustrations and actively solve problems. Random Experiments • An experiment is prescribed with N outcomes • finite {1 2 3} countable {1 2 3 …} uncountable {z>0} • A chance mechanism selects outcome – uncertainty • We have some knowledge of the likelihood of occurrence – often measured empirically through relative frequency in repeated independent trials • Kolmogorov developed consistent axiomatic framework for probability Relationship to Set Theory • Set Theory Probability • Universe • Elements • Algebra of subsets Sample space Outcomes Events • To each event the theory allows/assigns a number • Probability (event) indicates its likelihood of occurrence 0<= p <= 1 Set Operations – PPT Notation • Complement A^c = {c: c in S but c not in A} • Union (cup) A+B = { c: c in A or c in B } • Intersection (cap) AB = {c: c in A and c in B } • Often we use cap/cup/overbar in traditional math notation • MATLAB provides set operations as functions very useful when working with large sets Algebra 0f Events • Each experiment has outcomes in sample space S • Subsets of outcomes are called events. • The set of events must be an algebra –closure • • • • For events A,B,C, … then All complements are events A^c, B^c, … All finite unions & intersections are events All countable unions & intersections are events • For finite sample spaces the events consist of the set of all subsets. But too large for infinite S Probability Triple • (S, E, P) (Sample Space, Events, Probability) • Sample space – set of all possible outcomes • Events – family/collection of all subsets to which we assign probabilities. Subject to closure under set operations • Probability – maps events to unit interval [0,1] subject to 3 Axioms. For events A,B,C,… • (1) P(A) >= 0 non-negative probability • (2) P(S) = 1 something happens w probability 1 • (3) A, B disjoint P(A+B) = P(A) + P(B) • (3’) Can extend (3) to countable unions Examples • S = {T,H} = {0,1} 1 coin toss • E = { S^c = {}, {0}, {1}, S = {0,1} } • Given P(1) can compute probability of any e in E • S = {TT,HT,TH,HH} = {00,10,01,11} 2 coin toss • E = { S^c = {}, {0}, {1}, S = {0,1} } • Given P(1) can compute probability of any e in E • Each event can be given an intuitive name S = {00,10,01,11} {H=1,T=0} e 0 1 2 3 outcomes 1 0 0 0 0 Empty set = nothing happens 2 0 0 0 1 Exactly {11} “2 heads” 3 0 0 1 0 Exactly {01} “tail then head” 4 0 0 1 1 {01} or {11} “head on second toss} 5 0 1 0 0 Exactly {10} 1 0 0 1 Exactly “2 of the same” 1 1 1 0 “Anything but 2 heads” complement of #2 6 7 10 14 15 16 S = something happens Probability of an Event • Event is a subset of outcomes • An event is realized (occurs) if any of its members occurs • The outcomes of an experiment are by definition disjoint • P (event) is the sum of the probabilities of outcomes that define it • P(event) = P(outcome_1) + … • But events can contain many outcomes How to we know P • P( any event ) is known in the theory. But how? • Relative frequency in independent repeated trials • Uniformity considerations – insufficient reason to suspect otherwise “at random” • Subjective – expert opinion • What is the probability that the sun will rise tomorrow p=1? p<1? • What is the probability that you will obtain above 3.4 in EE416? • Engineering usage focuses on relative frequency • Based on past ( but relevant) history Theorems are derived from Axioms • P(A^c) = 1 - P(A) • P(S) = 1 P(empty set) = 0 • P(A+B) = P(A) + P(B) - P(AB) • P(A+B) < = P(A) + P(B) • A subset of B then P(A) <= P(B) • Proofs are instructive usually they relate to disjoint events Partitions • • • • If A us an event, and A in B in S (sample space) Let B_1, …B_N partition B Then P(A) = P(AB_1) + … +P(AB_N) • Simplifies more complicated events General Problem of Computing Prob • Given a combination of events, with knowing prob of each separately, determine prob of combination • Typical combinations “at least one” “k or more of” • Always can be written as set operations • Always can be enumerated, but can be a long list 1 Coin Toss • • • • • • • • Experiment – toss a coin S –{T,H} |S| =2 Here let the events be the power set Pwr(S) Set of all subsets with 2^2 -4 elements S^c = Empty set, H, T, {H or T} =S closed family Assign P(H) = p, p = 1/2 only for a fair coin Consistent with axioms if p in [0,1] & P(T) = 1-p Often insufficient reason to take p neq 1/2 Often p should be measured empirically, the fraction of heads in many trials (how many?) • The outcomes have prob {pT= 1-p, pH = p} Geometric Probability • In many board games spinners are used. You cut a circular region into n congruent pizza slices, and flip a spinner. It is obvious that the probability of any outcome is p = 1/n. • This can be argued to be the ratio of the arc length of the wedge to arc length of the circle • Geometric probabilities reduce to calculation of relative length/area/volume and are based on an “at random” assumption. This implies uniformity. 2 Coin Toss • S = {TT, TH, HT, HH} ordered (T1, T2) • Pwr(S) has 2^4 = 16 possible events • It is not possible to combine outcomes in any other way. Next assign a consistent probability to • (p1, p2,p3,p4) p_i in [0,1] sum p_i = 1 • Insufficient reason takes p_i = ¼ • But it depends on the random experiment (toss) • Allowable is any consistent set of probabilities. • Then eg, P(T1=TorT2=T) = P(HT + TH + TT) = =P(HT) + P(TH) + P(TT) Independence • • • • Two events A,B are independent when P(AB) = P(A)P(B) multiply probabilities This is a property of the probability assignment Examine the 2 coin toss in more detail • Dependent events – statistical relationship/linking • This allows for prediction of one given the other • Not a one-way causal relationship 2 Coin Toss • • • • • • • • • • Label the outcomes (00, 01, 10, 11) where T=0,H=1 Suppose we measured P(00) = 0.10 P(01) = 0.30 P(10) = 0.22 P(11) = 0.38 Are the tosses independent? P(t1=0)=P(00)+P(01) = 0 .1+0.3 =0.40 P(t1=1)= 1-.4 P(t2=0)=P(00)+P(10) = 0.1+0.22=0.32 P(t2=1 = 1- .32 Independence says that P(t1=0)P(t2=0) = P(00) but this does not hold! Under this P, Events are not independent! Are disjoint events independent? NO • Let A,B mutually exclusive AB = empty set • P(AB) = P(empty set) = 1-P(S) = 0 • They are not independent • Disjoint events are highly dependent – if one event occurs, the other cannot have occurred • This must be distinguished from independence • P(AB)=P(A)P(B) which might be zero if P(A)=0 • A property of the Prob Assnment, not the events Communications Networks • • • • A network is made up of nodes and links (edges) The link either work 1 or fail 0 with p=P(1) q=P(0) Links can be connected in various ways Find Probability P_N that a network works • Series Net: all links work P_N = p^N • Parallel Net: at least 1 link works P_N = 1 - q^N • Need to work through the derivation • Networks can combine series and parallel Target Detection • A radar takes N looks at a target, target is detected (when present) with p=0.9 • How does detection prob improve as we increase N • P_N := 1 –(1-p)^N but we need to explore numerically • For small N, • P = [ p, 2p-p^2, 3p -3p^2 +p^3, … ] • Is this a series or parallel problem? Birthday Problem • • • • • Birthdays occur at random on [1:365] no leap yr Sample r people’s birthdays, n=365 find P(hit) =P( 2or more birthdays in sample) P(no hit) = # ways to select wo rep/# ways w rep 1-P(hit)= (n)_r / n^r where (n)_r = n!/(n-r)! • Numerical implications are interesting Birthday Problem Numerics • Place r balls at random in n bins. What is the probability that 2 or more land in any bin • N=365, r=23 for a 50:50 chance of hits • Randomness causes clumping! • Table shows P(hit) vs r, number of balls placed • Balls can be calls, bins can be channels/frequencies 20 0.4 23 0.507 30 0.706 35 0.814 40 0.89 Birthday Problem Analysis • • • • • • • Give a simple expression for P(r,n) = 1 –(n)_r/n^r, when n is large Write Q = 1-P as a product Q=1-P(r,n) (n)_r/n^r = Prod_1^(r-1) [1-j/n] Go to a log form exp( sum of logs) Approximate the sum & simplify This is computational probability • Or just compute use Gamma functions Birthday Prob (n=365) Matlab Code • • • • • function birthdayprob( N, K); %function birthdaypob( N, K); %compute all birthday probabilities. kall =[0:K-1]; %Pnohit =(N)_k/N^k; • • • • • • Pnohit = exp( cumsum( log(1 -kall/N) ) ); Pnoapp = exp( -(kall+1).*(kall)/(2*N) ); % a very good approximation! X = [1:K]'*[ 1 1]; P1 = [ Pnohit;1-Pnohit]'; P2 = [ Pnoapp; 1-Pnoapp]'; • • • • • % note that we are plotting both tails figure(1); semilogy( X, P1,'-*'); hold on; semilogy( X, P2,'-o'); hold off title('Exact vs Approx hit Probabilities'); xlabel('Number of Balls Thrown');grid; Ranking Problems • You meet Alice, she has two brothers Bob and Chris. You don’t know anything more. • What is the probability that Alice is the oldest? • Use Insufficient Reason to determine • P(A oldest) = 1/3 • Now she tells you that Alice is older than Bob. • Determine P(A is oldest). • Hint: Consider the sample space uses outcomes are all possible rankings of age. (A>B>C, etc)