Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Probability & Random Variables Probability & Random Variables Prof. Jae Young Choi, Ph.D. Medical Imaging & Pattern Recognition Lab. Department of Biomedical Engineering Email: [email protected] URL: http://bprlab.tistory.com/notice/4 Jungwon University PROBABILITY • ◈ OBJECTIVES – 확률, 불규칙 변수와 신호의 이론과 응용에 대해 공부 – D/ Random Signal: • 어떤 확률적인 방식에 의해서만 그 특성이 결정되는 시간의 함수인 파 형 및 신호 – Ex) Broadcasting Radio Receiver • Voice = Desirable Waveform (Information) + Undesirable Waveform (Noise) = v(t) + n(t) – Ex) Television System • Picture Interference called “Snow” White Noise (백색잡음) PROBABILITY Random phenomena – Unable to predict the outcomes, but in the long-run, the outcomes exhibit statistical regularity. Examples 1. Tossing a coin – outcomes S ={Head, Tail} Unable to predict on each toss whether is Head or Tail. In the long run can predict that 50% of the time heads will occur and 50% of the time tails will occur PROBABILITY 2. Rolling a die – outcomes S ={ , , , , , } Unable to predict outcome but in the long run can one can determine that each outcome will occur 1/6 of the time. Use symmetry. Each side is the same. One side should not occur more frequently than another side in the long run. If the die is not balanced this may not be true. PROBABILITY 3. Rolling a two balanced dice – 36 outcomes 4. Stock market performance A stock currently has a price of $125.50. We will observe the price for the next 100 days typical outcomes 250 200 price – 150 100 50 0 0 20 40 60 day 80 100 ♣ MATHEMATICAL NOTATIONS • • • • • • • • • • • • • • • • • • A/ Ex/ D/ L/ T/ C/ →, ⇒ ↔, ⇔ ¬, A or Ac ∀ ∃ s.t. i.e. w/ p.t. Nbhd ∴ ∵ ASSUMPTION, AXIOM 가정, 공리 Example 예제 DEFINITION 정의 LEMMA 보조정리 THEOREM 정리 COROLLARY 결과정리 IMPLICATION …의미하다 NECESSARY & SUFFICIENT 필요충분조건 NOT, COMPLEMENT OF A 부정…, A의 여집합 FOR ALL 모든… THERE EXIST(S) …존재한다 SUCH THAT… 이러하다 id est (THAT IS) 즉, WITH, WHERE …과 함께, 여기서… POINT 점 NEIGHBORHOOD 주변, 이웃 THEREFORE 그러므로 BECAUSE 왜냐하면 The Greek Alphabet (그리스 자모 24개) The Greek Alphabet (그리스 자모 24개) 대문자 A B G D E Z H Q I K L M N X O P R S T U F C Y W 소문자 a b g d e z h q i k l m n x o p r s t u f c y w 명칭 비고 Alpha Beta Gamma Delta Epsilon Zeta Eta Theta Iota Kappa Lambda Mu Nu Xi Omicron Pi Rho Sigma Tau Upsilon Phi chi Psi Omega A, a B, b G, g D, d E, e Z, z H, h Q, q I, i K, k L, l M, m N, n X, x O, o P, p R, r S, s T, t U, u F, f C, c Y, y W, w PROBABILITY • ♣ SET DEFINITIONS – D/ Set : – D/ Elements : – D/ Class : a collection of objects A objects in a set , a A, e A w/ A a, b, c, d a collection of sets {A, B, C} – ☞ Two methods in specifying a set: • ① Tabular Method : explicitly enumerated elements Ex) A = {6, 7, 8, 9} • ② Rule Method : implicitly described by some rules Ex) B = {5 < b < 10, b = integers} PROBABILITY • ♣ SET DEFINITIONS – D/ Countable Set : • its elements 1-to-1 correspondence with the natural numbers (opp. Uncountable Set) – D/ Empty Set (Null Set) : f • a set w/o any elements – D/ Finite Set : • a set of either countable or f w/ a finite number of elements (opp. Infinite Set) – ex) a countably infinite set the set of the natural numbers, N = {1, 2, 3, …} PROBABILITY • ♣ SET DEFINITIONS – D/ Subset A of B : • A is contained in B A B – D/ Proper Subset A of B : • ∃ at least one element in B, NOT in A A B A B – D/ Mutually Exclusive (M.E., Disjoint) sets : • Absolutely no common elements between A and B A B f – D/ Universal Set S : • The largest or all-encompassing set of elements A B S PROBABILITY • ♣ SET DEFINITIONS • Ex) A = {1, 3, 5, 7} tabular표시, B = {1, 2, 3, …} tabular표시, C = {0.5 < c ≤ 8.5} rule표시, D = {0.0} tabular표시, E = {2, 4, 6, 8, 10, 12} tabular표시, F = {-5.0 < f ≤ 12.0} rule표시, A B, C , F C, D F EB countably finite countably infinite uncountably infinite countably finite countably finite uncountably infinite A, D, E ~ M.E. Sets • Ex) A problem of rolling a die (주사위 던지기) – The universal set, S = {1, 2, 3, 4, 5, 6} – (i) a person wins if odd : – (ii) another person wins if less than 5 : A = {1, 3, 5} B = {1, 2, 3, 4} – Let #S = n, then #{all the possible subsets} = 2n. – Therefore, if n = 6, 26 = 64 ways of defining “winning”w/ one die. PROBABILITY • ♣ SET OPERATIONS – ◆ Venn Diagram (집합의 기하학적 도식법) • B is a subset of A B A • C is M.E. with both A and B – ▶ Equality and Difference • D/ Equality : • D/ Difference : A B A B and B A A B A B – Ex) Let A = {0.6 < a ≤ 1.6}, B = {1.0 ≤ b ≤ 2.5} A - B = {0.6 < c < 1.0}, B - A = {1.6 < d ≤ 2.5} Note : A B B A PROBABILITY • ♣ SET OPERATIONS – ▶ Union and Intersection • D/ Union of 2 sets, A and B (Sum of 2 sets) : C A B n in general, C A1 A2 An Ai i 1 • D/ Intersection of 2 sets, A and B (Product of 2 sets) : D A B n in general, D A1 A2 An Ai i 1 A B f – Ex) If A and B are M.E., then – ▶ Complement • D/ Complement of a set A : ASA – Ex) Universal Set vs. Empty Set :f S , S f A A S, A A f PROBABILITY • ♣ SET OPERATIONS – Ex) Integer Subsets : S = {x| x is a natural number} • A = {1, 3, 5, 12}, B = {2, 6, 7, 8, 9, 10, 11}, C = {1, 3, 4, 6, 7, 8} A B f , B C 6,7,8, C A 1,3 A B 1,2,3,5,6,7,8,9,10,11,12, etc... PROBABILITY • ♣ SET ALGEBRA – T/ Commutative Law (교환법칙) :A B B A A B B A – T/ Distributive Law (분배법칙) : A B C A B A C A B C A B A C – T/ Associative Law (결합법칙) : A B C A B C A B C A B C A B C A B C – T/ De Morgan’s Law (드 모르강의 법칙) : A BA B A BA B – ☞ Duality Principle : • As shown above, the laws preserved by changing S into f, union op. into intersection op., and vice versa PROBABILITY • ♣ PROBABILITY INTRODUCED THROUGH SETS Trial (시행) • • • • • • {Physical} Experiment Outcome (결과) Experiments (실험) Sample Space (표본공간) Events (사건, 사상 ~ 정성적) = Sets (집합 ~ 정량적) Discrete Sample Space (이산적 표본공간) Continuous Sample Space (연속적 표본공간) Mathematical Model of Experiments ~ Probability를 사용 The sample Space, S The sample space, S, for a random phenomena is the set of all possible outcomes. The sample space S may contain 1. A finite number of outcomes. 2. A countably infinite number of outcomes, or 3. An uncountably infinite number of outcomes. A countably infinite number of outcomes means that the outcomes are in a one-one correspondence with the positive integers {1, 2, 3, 4, 5, …} This means that the outcomes can be labeled with the positive integers. S = {O1, O2, O3, O4, O5, …} A uncountably infinite number of outcomes means that the outcomes are can not be put in a one-one correspondence with the positive integers. Example: A spinner on a circular disc is spun and points at a value x on a circular disc whose circumference is 1. 0.0 0.9 0.1 S = {x | 0 ≤ x <1} = [0,1) x 0.2 0.8 0.0 0.3 0.7 0.6 0.4 0.5 [ S 1.0 ) Examples 1. Tossing a coin – outcomes S ={Head, Tail} 2. Rolling a die – outcomes S ={ , , , , , ={1, 2, 3, 4, 5, 6} } 3. Rolling a two balanced dice – 36 outcomes S ={ (1, 1), (1, 2), (1, 3), (1, 4), (1, 5), (1, 6), (2, 1), (2, 2), (2, 3), (2, 4), (2, 5), (2, 6), (3, 1), (3, 2), (3, 3), (3, 4), (3, 5), (3, 6), (4, 1), (4, 2), (4, 3), (4, 4), (4, 5), (4, 6), (5, 1), (5, 2), (5, 3), (5, 4), (5, 5), (5, 6), (6, 1), (6, 2), (6, 3), (6, 4), (6, 5), (6, 6)} outcome (x, y), x = value showing on die 1 y = value showing on die 2 An Event , E The event, E, is any subset of the sample space, S. i.e. any set of outcomes (not necessarily all outcomes) of the random phenomena S E The event, E, is said to have occurred if after the outcome has been observed the outcome lies in E. S E Examples 1. Rolling a die – outcomes S ={ , , , , , } ={1, 2, 3, 4, 5, 6} E = the event that an even number is rolled = {2, 4, 6} ={ , , } 2. Rolling a two balanced dice – 36 outcomes S ={ (1, 1), (1, 2), (1, 3), (1, 4), (1, 5), (1, 6), (2, 1), (2, 2), (2, 3), (2, 4), (2, 5), (2, 6), (3, 1), (3, 2), (3, 3), (3, 4), (3, 5), (3, 6), (4, 1), (4, 2), (4, 3), (4, 4), (4, 5), (4, 6), (5, 1), (5, 2), (5, 3), (5, 4), (5, 5), (5, 6), (6, 1), (6, 2), (6, 3), (6, 4), (6, 5), (6, 6)} outcome (x, y), x = value showing on die 1 y = value showing on die 2 E = the event that a “7” is rolled ={ (6, 1), (5, 2), (4, 3), (3, 4), (3, 5), (1, 6)} Special Events The Null Event, The empty event - f f = { } = the event that contains no outcomes The Entire Event, The Sample Space - S S = the event that contains all outcomes The empty event, f , never occurs. The entire event, S, always occurs. Set operations on Events Union Let A and B be two events, then the union of A and B is the event (denoted by A B ) defined by: AB A B = {e| e belongs to A or e belongs to B} AB A B A B The event A B occurs if the event A occurs or the event and B occurs . A B AB A B The event A B occurs if the event A occurs and the event and B occurs . AB A B A B Complement Let A be any event, then the complement of A (denoted by A ) defined by: A = {e| e does not belongs to A} A A The event Aoccurs if the event A does not occur A A In problems you will recognize that you are working with: 1. Union if you see the word or, 2. Intersection if you see the word and, 3. Complement if you see the word not. DeMoivre’s laws 1. A B A B = 2. A B A B = DeMoivre’s laws (in words) 1. A B A B The event A or B does not occur if the event A does not occur and the event B does not occur 2. A B A B The event A and B does not occur if the event A does not occur = or the event B does not occur Rules involving the empty set, f, and the entire event, S. 1. A f A 2. A f f 3. A S S 4. A S A Definition: mutually exclusive Two events A and B are called mutually exclusive if: A B f A B If two events A and B are are mutually exclusive then: 1. They have no outcomes in common. They can’t occur at the same time. The outcome of the random experiment can not belong to both A and B. A B Some other set notation We will use the notation e A to mean that e is an element of A. We will use the notation e A to mean that e is not an element of A. Thus A B e e A or e B A B e e A and e B A e e A We will use the notation A B (or B A) to mean that A is a subset B. (B is a superset of A.) i.e. if e A then e B. B A Union and Intersection more than two events k Union: i 1 Ei E1 E2 E3 Ei E1 E2 E3 i 1 E2 E1 E3 Ek k Intersection: i 1 Ei E1 E2 E3 Ei E1 E2 E3 i 1 E2 E1 E3 Ek DeMorgan’s laws 1. 2. i Ei i Ei = Ei i Ei i PROBABILITY • ♣ PROBABILITY INTRODUCED THROUGH SETS Trial (시행) {Physical} Experiment Outcome (결과) – D/ Probability Definitions and 3 Axioms • A1/ • A2/ • A3/ P A 0 Probability of an event A is greater or equal to zero. PS 1 Probability of a sample space is equal to one. n n P Ai P Ai i 1 i 1 if Ai A j f , i j i.e., M.E. Events n possible PROBABILITY • ♣ JOINT & CONDITIONAL PROBABILITY – D/ Joint Probability (결합확률) : P A B if, for two events, A and B, ∃common elements in A B – T/ 1. P A B P A PB P A B 2. P A B P A PB P A B P A PB w/ equality holds if (A, B ~ M.E. events) A B f A A B B Example for Sample Space and Event PROBABILITY • ♣ JOINT & CONDITIONAL PROBABILITY – D/ Conditional Probability (조건확률) : P A | B • For a nonzero prob. P(B) >0, cond. prob. of an event A given B is P A B P A | B P B • Reduced Sample Space S’= B P A B PBP A | B • T/ if A and B A B f are M.E. then P A | B Pf / PB 0 • T/ if A and C are M.E. then P A C | B P A | B PC | B Example for Conditional Probability PROBABILITY • ♣ TOTAL PROBABILITY – D/ Total Probability, P(A) (총확률) : • The probability P(A) of any event A can be expressed in terms of n conditional probabilities of M.E. events Bi’s whose union equals the sample space S. • (i) Bi’s ~ M.E. events : Bi B j f i j n • (ii) Union of Bi’s ~ Sample Space : Bi S i 1 PROBABILITY • ♣ TOTAL PROBABILITY – D/ Total Probability, P(A) n • Total Probability of Event A is P A P A | Bi PBi i 1 • Proof/ n n A A S A Bi A Bi i 1 i 1 n n P A P A Bi P A Bi i 1 i 1 n P A | Bi PBi i 1 by Distr. Law, by Axiom 3, A Bi ' s ~ M.E. by Cond. Prob. Q.E.D. PROBABILITY we already know this !! • ♣ BAYES’THEOREM – ☞ Problem Statement : • How to find “Inverse Probability”? PBi | A P A Bi P A if P A 0 P A | Bi P A Bi P Bi if P Bi 0 PBi | A P A | Bi inverse relation • Inverse Probability : P A Bi P A | Bi PBi P A P A P A | Bi PBi P A | B1 PB1 P A | Bn PBn PBi | A by cond. prob. by total prob. PROBABILITY • ♣ INDEPENDENT EVENTS (독립 사건) – D/ Statistical Independence (통계적 독립성, S.I.): • For 2 events, A and B, let P(A), P(B) > 0, if P(A) is NOT affected by the other event B, or vice versa, i.e., P(A|B) = P(A) and P(B|A) = P(B), then P A B P A | BPB PB | AP A P APB P A B P APB P, A B 0 A B f • NOTE : For M.E. events, A and B This means P(A)P(B) = 0 for M.E. events to be independent, which is NOT possible because P(A) >0 and P(B) >0. Therefore, it CANNOT be both M.E. P A1 and An S.I. P A1 P An • Generalization : PROBABILITY • ♣ BERNOULLI TRIALS (베르누이 시행) – D/ Bernoulli Trials = Repeated Independent Experiments • Ex) Flipping a coin, Pass/Fail an exam, Hit/Miss a target, Win/Lose a game, Receiving 0 or 1 in a bit-stream, etc. • Let A be an elementary event of repeated independent trials – P(A) = p ~ probability of a success – P(Ac) = 1- p = q ~ probability of a failure – #Repeated Trials = n, #Success = k Then, #instances = binomial ncoefficients k nk P A occurs exactly k times p 1 p k n n! k k!(n k )! PROBABILITY • ♣ Examples • Ex 1) 100 registors in a box Tolerance Resistance (Ω) 5% 10% Total 22 10 14 24 47 28 16 44 100 24 8 32 – Event A =‘Draw a 47Ω registor’ – Event B =‘Draw a registor w/ 5% tolerance’ – Event C =‘Draw a 100Ω registor’ – (1) Find probabilities : P(A), P(B), and P(C) P(A) = P(47Ω) = 0.44, P(B) = P(5%) = 0.62, P(C)=P(100Ω) = 0.32 – (2) Find all joint probabilities : P(A∩B), P(A∩C), and P(B∩C) P(A∩B)=P(47Ω∩5%)=0.28, P(A∩C)=P(47Ω∩100Ω)=0, P(B∩C)=P(5%∩100Ω)=0.24 – (3) Find conditional probabilities : P(A|B), P(A|C), and P(B|C) P(A|B) = P(A∩B)/P(B) = 28/62, P(A|C) = P(A∩C)/P(C) = 0, P(B|C) = P(B∩C)/P(C) = 24/32 PROBABILITY • ♣ Examples • Ex 2) Elementary Binary Communication Systems {0, 1} 신호를 채널을 통해 전달하는 체계 » » » » B1 = Event ‘The Symbol B0 = Event ‘The Symbol A1 = Event ‘The Symbol A0 = Event ‘The Symbol before the Channel is 1’ before the Channel is 0’ after the Channel is 1’ after the Channel is 0’ – A/ ‘Binary Symmetrical Channel’(이진 대칭 채널) » Selection Probability of 0 or 1 for transmission : P(B1)=0.6, P(B0)=0.4 » Reception Probability given a 1 was transmitted : P(A1|B1)=0.9, P(A0|B1)=0.1 » Reception Probability given a 0 was transmitted : P(A1|B0)=0.1, P(A0|B0)=0.9 » cf. P(A1|Bi) + P(A0|Bi) = 1 ∵ A1, A0 are M.E. PROBABILITY • ♣ Examples • Ex 2) Elementary Binary Communication Systems – ▷ Received Symbol Probabilities : (측정된 출력확률은 총확률로 구함 ) » P(A1) = P(A1|B1)P(B1) + P(A1|B0)P(B0) = 0.9×0.6 + 0.1×0.4 = 0.58 » P(A0) = P(A0|B1)P(B1) + P(A0|B0)P(B0) = 0.1×0.6 + 0.9×0.4 = 0.42 – ▷ Inverse Probabilities : (결과로 나타나는 사후확률은 역확률로 구함) » P(Bk|Ak) = a posteriori probabilities, cf. P(Bk) = a priori probabilities (사후확률, 후험적확률) (사전확률, 선험적확률) » » » » P(B1|A1) = P(B0|A0) = P(B1|A0) = P(B0|A1) = P(A1|B1)P(B1)/P(A1) ≒ P(A0|B0)P(B0)/P(A0) ≒ P(A0|B1)P(B1)/P(A0) ≒ P(A1|B0)P(B0)/P(A1) ≒ 0.931 0.857 0.143 0.069 →‘Correct’ → 〃 → ‘Error’ → 〃 PROBABILITY • ♥ MORE on PROBABILITY • T/ [Fundamental Principle] A에 대한 서로 다른 연산이 m개, B에 대한 서로 다른 연산이 n개일 때, A와 B에 대한 서로 다른 연산의 경우의 수는 모두 m×n 가지이다. • Ex) PASSWORD : 알파벳 2개와 숫자 2개로 이루어진 가능한 패스워 드의 개수는? #Letter #Letter #Number #Number = 26 × 26 × 10 × 10 = 67,600 • T/ [Stirling's Formula] = Approximation of n! n! 2p n n ½ e n log n! ½ log 2p n ½log n n log e PROBABILITY • ♥ MORE on PROBABILITY • T/ [Permutation] → Permutation w/o Repetition : 서로 다른 n개 의 객체로부터 중복을 허용하지 않고, 순서를 고려하여 r개를 나열하 는 경우의 수 n Pr n! (n r )! Ex) 4×4 Puzzle을 나열하는 가지수는? 16! = 20,922,789,888,000 가지 • T/ [Maxwell-Boltzmann Distribution (M-B)] → Permutation w/ Repetition : 서로 다른 n개의 객체로부터 중복을 허용하고, 순서를 고 려하여 r개를 나열하는 경우의 수 nr PROBABILITY • ♥ MORE on PROBABILITY Ex) DNA Coding에서 4개의 Key Nucleotides {A (Adenine), G (Guanine), C (Cytosin), U (Urasil)}를 사용하여 20가지의 아미노산이 형성되었을 때, 이를 구별할 수 있는 Neucleotide의 최소 길이는? n = 4 이므로 4r ≥ 20 에서 r ≥ log(20)/log(4) r = 3 • T/ [Fermi-Dirac Distribution (F-D)]→ Combination w/o Repetition : 서로 다른 n개의 객체로부터 중복을 허용하지 않고, 순 nr개의 n! 서에 무관하게 조합을 C 만드는 경우의 수 r Ex) n r r!(n r )! 8 8! 8 C2 28 6! APEC에서 지도자가 2 8명의2!정치 만나서 서로 악수하는 경우의 수 ? PROBABILITY • ♥ MORE on PROBABILITY • T/ [Bose-Einstein Distribution (B-E)]→ Combination w/ Repetition: 서로 다른 n개의 객체로부터 중복을 허용하고, 순서에 무관하게 r개의 조합을 만드는 경우의 수 n r 1 (n r 1)! n r 1 Cr r r!(n 1)! Ex) 3개 변수를 갖는 Analytic Function f의 서로 다른 6차 편미분을 구하 6 f ( x, y, z ) 는 x, x, y, y, y, z x 2 y 3z 방법 ? 8! 28들면, r=6 이고 6 y, 1 Cz에 6 대해 예를 n=3이므로3x, 6!2! PROBABILITY • ♥ MORE on PROBABILITY Distinct or Distinguishable (Ordered) ↓ n Pr Indistinct or Indistinguishable (Not Ordered) ↓ Fermi-Dirac nr Maxwell-Boltzmann M-B ← No Repetition (No Replacement) Cr n F-D ← Repetition (Replacement) Cr n+r-1 Bose-Einstein B-E PROBABILITY • ♡ PROGRAM PROJECT#1 : CARD DECK 52장이 있다. – ◆ A 2 3 4 5 6 7 8 9 10 J Q K – ♣ A 2 3 4 5 6 7 8 9 10 J Q K – ♥ A 2 3 4 5 6 7 8 9 10 J Q K – ♠ A 2 3 4 5 6 7 8 9 10 J Q K Random하게 선택하여 5장을 뽑는 데 (한번 나온 카드는 안 나오도록) □□□□□ (1) Shuffling Program (2) 1장씩 돌아가며 총 5장이 되도록 4인에게 분배하는 Program [Use Uniform Distribution in Your PC, rand() in Matlab or random() in C/C++]