Download Sample space

Chapter 2 Concepts of Prob. Theory
•
Random Experiment: an experiment in which outcome
varies in an unpredictable fashion when the experiment is
repeated under the same condition
Specified by:
1. An experimental procedure
2. One or more measurements or observations
tch-prob
1
EXAMPLE 2.1 :
Experiment E1 : Select a ball form an urn containing balls numbered 1 to 50.
Note the number of the ball.
Experiment E2 : Select a ball form an urn containing balls numbered 1 to 4.
Suppose that balls 1 and 2 are black and that balls 3 and 4 are white. Note
number and color of the ball you select.
Experiment E3 : Toss a coin three times and note the sequence of heads/ tails.
Experiment E4 : Toss a coin three times and note the number of heads.
Experiment E6 : A block of information is transmitted repeatedly over a noisy
channel until an error-free block arrives at the receiver.
Experiment E7 : Pick a number at random between zero and one.
Experiment E12 : Pick two numbers at random between zero and one.
Experiment E13 : Pick a number X at random between zero and one, then
pick a number Y at random between zero and X.
tch-prob
2
Discussions
• Compare E3 and E4: same procedure, different
observations
• A random experiment with multiple measurements
or observations: E2, E3, E12, E13
• sequential experiment consists of multiple
subexperiments: E3, E4, E6, E12, E13
• dependent subexperiments: E13
tch-prob
3
Sample Space
• Outcome, sample point: 
– one and only one per experiment
– mutually exclusive, cannot occur simultaneously
• Sample space: set of all possible outcomes
• Example 2.2: sample spaces
•
S1, S2 , S3, S6 , S7 , S12 , S13
number of outcomes:
– Finite
– countably infinite S6
– unaccountably infinite S7






tch-prob
Discrete
sample space
Continuous
Sample space
4
EXAMPLE 2.2 :
The sample spaces corresponding to the experiments in Example
2.1 are given below using set notation :
S1  1, 2,......,50
S2  1, b  ,  2, b  ,  3, w  ,  4, w 
S3  HHH , HHT , HTH , THH , TTH , THT , HTT , TTT 
S4  0,1, 2,3
S6  1, 2,3,

S7   x : 0  x  1   0,1
S12 
 x, y  : 0  x  1 and

0  y 1
S13   x, y  : 0  y  x  1
tch-prob
5
Event: a subset of S, a collection of outcomes that satisfy certain conditions
Certain event: S
null event: 
elementary event: event of a single outcome
E ,E ,E
2 4 7
A2: The ball is white and even-numbered
Example 2.3:
A4: The number of heads equals the number of tails
A7: The number selected is nonnegative
tch-prob
6
Set Operations
1.
2.
Union
A B
A
Intersection A B
mutually exclusive
3.
B
Complement
imply
A B
equal
A=B
A B 
Ac  S  A
- Commutative properties of set operations
A BB
A
A BB
- Associative properties of set operations
A ( B C )  ( A B) C
- Distributive properties of set operations
A ( B C )  ( A B) ( A C )
- De Morgan’s Rules
Venn diagrams:
( A B)c  Ac
Bc
( A B)c  Ac
A
A ( B C )  ( A B) C
Bc
Fig. 2.2
tch-prob
7
Example 2.5: Configuration of a three-component system
a. series
all three
C1
C2
C3
A1 I A2 I A3
b. parallel
at least one of three
A1
A2
C1
C2
A3
C3
c. two-out-of-three
( A1
A2
A3 ) ( A1c
n
A  A A ..... An
k 1 k 1 2
n
A  A A ..... An
k 1 k 1 2


A
A
k
k 1 k
k 1
A2
A3 ) ...
Ak : event that component k is functioning
tch-prob
8
2.2
The Axioms of Probability
A probability law for the experiment E is a rule that assigns to each
event a number P(A), called the probability of A, that satisfies the
following axioms:
Axiom I:
Axiom II:
0  P( A)
nonnegative
P(S)=1
Axiom III: If
total=1
A B  , then
P  A B   P  A  P  B 
Axiom III’: If Ai A j  for all i  j , then P   A    P  A 
  k 


k
 k 1
 k 1
A set of consistency rules that any valid probability assignment must satisfy.
tch-prob
9
Corollary 1. P  Ac  1 P  A


pf:
A Ac 
1 P  S   P  A Ac   P  A  P  Ac 
Corollary 2. P  A 1
pf: from Cor.1,
Corollary 3. P    0
pf: Let A=S,
Corollary 4.
P  A 1 P  Ac  1
Ac 
in Cor.1.
A , A ,...., An, are pairwise mutually exclusive,
1 2
then  n  n  
P
A    P A 
for n  2
 k 1 k  k 1  k 
tch-prob
10




Corollary 5. P  A B   P  A  P  B   P  A B 
pf:
P  A B   P  A Bc   P  B Ac   P  A B 
P  A  P  A Bc   P  A B 




 
P  B   P  B Ac   P  A B 




A
B
Corollary 6.
P




n  
n1


A   P  A    P  A A   ....  1 P  A .... An 
j
j k 
 1

k 1 k j1   jk 
n




Corollary 7. If A  B , then P  A  P  B 
pf. P  B   P  A  P  Ac B   P  A


 
 
 
tch-prob
11
Axioms + corollaries  provide rules for computing the probability of
certain events in terms of other events. However, we still need
initial probability assignment
1.
Discrete Sample Spaces
find the prob.of elementary events;
all distinct elementary events are mutually exclusive,
Example 2.6., 2.7
2. Continuous Sample Spaces
assign prob. to intervals of the real line or rectangular regions in the
plane
Example 2.11
tch-prob
12
Discrete Sample Spaces
First, suppose that the sample space is finite,
and is given by
S  a1 , a2
an


P  B   P  a1' , a2' ,
 


, am' 
 
 P  a1'   P  a2'  
(by corollary ?)
 
 P  am' 
(2.9)
If S is countably infinite, then Axiom Ⅲ’ implies that the
probability of an event such as D  b1' ,b2'  is given by

 
 
P  D  P  b1'   P  b2'  

(2.10)
tch-prob
13
If the sample space has n elements, S  a1 ,an , a probability
assignment of particular interest is the case of equally likely
outcomes. The probability of the elementary events is
Pa1  Pa2     Pan  
1
n
(2.11)
The probability of any event that consists of k outcomes, say
B  a1' ,  ak' , is


k
PB   Pa    Pa  
n
'
1
'
k
2.12 
Thus, if outcomes are equally likely, then the probability of
an event is equal to the number of outcomes in the event
divided by the total number of outcomes in the sample space.
(remember the classical definition?)
tch-prob
14
EXAMPLE 2.6
An urn contains 10 identical balls numbered 0,1,…, 9. A random
experiment involves selecting a ball from the urn and noting the
number of the ball. Find the probability of the following events :
A = “number of ball selected is odd,”
B = “number of ball selected is a multiple of 3,”
C = “number of ball selected is less than 5,”
and of A  B and A  B  C .
tch-prob
15
The sample space is S  0,1,,9 , so the sets of outcomes
corresponding to the above events are
A  1,3,5,7,9,
B  3,6,9 , and C  0,1,2,3,4.
If we assume that the outcomes are equally likely, then
PA  P1  P3  P5  P7  P9 
PB   P3  P6  P9 
5
10
3
10 .
PC   P0  P1  P2  P3  P4 
5
.
10
From Corollary 5.
5 3 2
6
PA  B   PA  PB   PA  B      .
10 10 10 10
tch-prob
16
where we have used the fact that A  B  3,9, so PA  B  2 10.
From Corollary 6,
PA  B  C  PA  PB  PC  PA  B
P  A  C   P  B  C   P  A  B  C 
5 3 5 2 2 1 1
     
10 10 10 10 10 10 10
9

10

tch-prob
17
EXAMPLE 2.7
Suppose that a coin is tossed three times. If we observe the
sequence of heads and tails, then there are eight possible outcomes
. If we
S  HHH , HHT , HTH , THH , TTH , THT , HTT , TTT 
assume that the outcomes of S3 are equiprobable, then the
probability of each of the eight elementary events is 1/8. This
probability assignment implies that the probability of obtaining two
heads in three tosses is, by Corollary 3,


P "2 heads in 3 tosses"  PHHT , HTH , THH 
 PHHT   PHTH   PTHH  
3
8
*If we count the number of heads in three tosses, then S  0,1, 2,3


P "2 heads in 3 tosses"  P2 
tch-prob
1
4
18
Continuous Sample Spaces
EXAMPLE 2.11
Consider Experiment E12, where we picked two number x and y at
random between zero and one. The sample space is then the unit
square shown in Fig. 2.8(a). If we suppose that all pairs of numbers
in the unit square are equally likely to be selected, then it is
reasonable to use a probability assignment in which the probability of
any region R inside the unit square is equal to the area of R. Find the
probability of the following events: A  x  0.5, B  y  0.5 , and
C  x  y .
Figures 2.8(b) through 2.8(c) show the regions corresponding to
the events A, B, and C. Clearly each of these regions has area ½.
1
1
1
Thus
PA 
PB  
PC  
2
2
2
tch-prob
19
FIGURE 2.8
a Two-dimensional sample space and three events.
y
y
x
S
1
2
x
x
0
0
1
(a) Sample space
1
(b) Event
y
1

x  
2

y
y
1
2
x y
0
(c) Event
1
x
x
0
1
(d) Event x  y
1

y  
2

tch-prob
20
2.3 Computing probabilities using counting methods
In many experiments with finite sample space, the outcomes can be
assumed to be equiprobable.
Prob. 
1.
Counting the number of outcomes in an event
Sampling with Replacement and with Ordering
choose k objects from a set A that has n distinct objects, with replacement
and note the order. The experiment produces an ordered k-tuple
x
x
x
1, 2,..., k

where xi  A and i 1,2,..., k
The number of distinct ordered k-tuples = nk
Ex.2.12
5 balls, 1,2,3,4,5 select 2
52  25
Pr[2 balls with same number] = 5
25
tch-prob
21
2.
Sampling without Replacement and with ordering
choose k objects from a set A that has n distinct objects,
without replacement. k  n
number of distinct ordered k-tuples = n (n-1)…(n-k+1)
Ex.2.13
5 balls, select 2
5 4  20
Pr [1st number >2nd number ]= 10/20
tch-prob
22
Permutations of n Distinct Objects
Consider the case when k=n
number of permutations of n objects = n!
For large n, stirling’s formula is useful
n1
n! 2 n 2en
Ex. 2.16
12 balls randomly placed into 12 cells, where more than 1 ball is
allowed to occupy a cell.
Pr [ all cells are occupied ]=
12! 12
5
5.37

10
~
12
tch-prob
23
3.
Sampling without Replacement and without ordering
n
~ ” combination of size k” = Ck
each combination has k! possible orders
C n  k ! n  n 1.... n  k 1
k
n  n 1.... n  k 1
 n
C 
 n!   
k
k!
k ! n  k !  k 
n
 
  “binomial coefficient”, read “n choose k”
k 
n
Ex. 2.18 The number of distinctive permutation of k white balls
and n-k black balls
n!
k ! n  k !
tch-prob
24
Ex. 2.19 A batch of 50 items contains 10 defective items.
Select 10 items at random, Pr [ 5 out of 10 defective ]=?





10   40 


5   5 
 50 




 10 


Exercise prob. 47: Multinomial
Ex.2.20
Toss a die 12 times. Pr [ each number appears twice ]= ?
12!
2!2!2!2!2!2!
612

{1,1,2,2,3,3, …….6,6} permutation
tch-prob
25
4. Sampling with Replacement and without ordering





n 1 k   n 1 k 


k   n 1 
Pick k balls from an urn of
n balls. (k may be greater than n)
Suppose k=5, n=4
object
1
2
XX
3
4
X
XX
The result can be summarized as XX // X / XX
8
8!
=
k: x
  
3!5!  3 
n-1: /
k  n 1!

= k ! n 1 !
 
Ex.
Place k balls into n cells.
tch-prob
26
2.4 Conditional Probability
We are interested in knowing whether A and B are related, i.e. ,
if B occurs, does it alter the likelihood of A occurs.
Define the conditional probability as
B
P  A| B  


P AI B
P  B 
- renormalize


A
for P[B]>0
P[ A I B ]
tch-prob
27
Example 2.21. A ball is selected from an urn containing 2 black balls (1,2)
and 2 white balls (3,4).
A: Black ball selected
½
B: even-numbered ball
½
C: number > 2
½
P  A I B  0.25
P  A B  

 0.5  P  A


0.5
P  B 
P  A I C 
P AC 
 0  0  P  A


0.5
P C 






tch-prob
Not related
Related
28
P  AI B   P  B  P  A B  is useful in finding prob. in sequential experiments
 P  A P  B A
Ex.2.22 An urn contains 2 black, 3 white balls.
Two balls are selected without replacement.
Sequence of colors is noted.
Sol. 1. b b
two subexperiments
B1
bw
W1
2/5
3/5
wb
B2
ww
1/4
P[B1 I B2 ]  P[B1]P[B2 / B1]
tch-prob
1/10
W2
3/4
3/10
B2
W2
2/4
3/10
2/4
3/10
Tree diagram
29
Let B1 , B2 , , Bn be mutually exclusive, whose union = S ,
S
partition of S.
A  A I S  A I (B1 U B2 UU Bn )
  A I B1  U A I B2  U...U AI Bn 

 



B1
B2
A
Bn
P  A  P  AI B1   P  AI B2  ... P  AI Bn 
“ Theorem on total probability”
P  A  P  B1  P  A B1   P  B2  P  A B2  ... P  A Bn  P  Bn 
Ex. 2.24 In prev. example (2.22) find P[W2]=Pr [second ball is white]
P W2   P W2 B1  P  B1   P W2 W1  P W1   3  2  1  3  3
4 5 2 5 5
tch-prob
30
Suppose event A occurs, what is the prob. of event Bj?
Bayes’ Rule
P  B j

 AI B 
 A B  P B 
P
P



j 
j 
A     j   n 

P  A
 P  A Bk  P  Bk 
k 1
Before experiment: P[ Bj ]= a priori probability
After experiment: A occurs
P[ Bj/A ]= a posteriori probability
tch-prob
31
Ai
Ex.2.26. Binary symmetric channel
1-p
0
Suppose P  A   P  A   1
 0
 1 2
1-e
Bj
0
e
e
p
1
1-e
1
P  B1   P  B1 A0  P  A0   P  B1 A1   A1   e 1  1 e  1  1
2
2 2
B A  P A 
P
e/2
P  A0 B1    1 0    0   1/
 e,
2
P  B1 
P  A1 B1   1 e 
tch-prob
32
2.5 Independence of Events
If knowledge of the occurrence of an event B does not alter the probability
of some other event A, then A is independent of B.
P  A I B 
 A
P  A B  

P
 
P  B 
Two events A and B are independent if
P  AI B   P  A P  B 
tch-prob
33
Ex. 2.28. A ball is selected from an urn of 4 balls
{1,b},{2,b},{3,w},{4,w}
A: black ball is selected
B: even-numbered ball
C: number >2
P[A]=P[B]=P[C]=0.5
P  AI B   1  P  A P  B 
4
P  AI C   0
P  A B   P  A
P  A C   P  A
If two events have non zero prob., and are mutually exclusive, then they
cannot be independent.
0  P  AI B   P  A P  B   0
tch-prob
34
Three events A, B, and C are independent if
P  A I B   P  A P  B 
P  A I C   P  A P C 
P  B I C   P  B  P C 
P  A I B I C   P  A P  B  P C 
Ex. 2.30.
P  B I D   1  P  B  P  D 
4
P  B I F   1  P  B  P  F 
4
P  D I F   1  P  D  P  F 
4
P  B I D I F   0  P  B  P  D  P  F 
B
F
D
F
Common application of the independence concept is in making the assumption
that the event of separate experiments are independent.
Ex.2.31. A fair coin is tossed three times. P[HHH]=1/8, ….
tch-prob
35
2.6. Sequential Experiments
• Sequence of independent experiments
If sub experiments are independent, and event Ak concerns outcomes of
the kth subexperiment, then A1 , A2 , , An
are independent.
P  A1 I A2 I I An   P  A1  P  A2 ...P  An 
• Bernoulli trial:
Perform an experiment once, note whether an event A occurs.
success, if A occurs
prob= p
failure, otherwise.
prob= 1-p
Pn(k)= prob. of k successes in n independent repetitions of a Bernoulli trial = ?
tch-prob
36
Ex.2.34. toss a coin 3 times.
P  k  0 
P[ H ]= p
P  k 2  3 p2 1 p 
1 p 
3
P  k 1  3 p 1 p 
2
P  k 3  p3
Binomial probability law
pn  k  




n
k
 pk


1 p 
nk
k=0,1,…,n
tch-prob
37
Binomial theorem
n  n  k nk
n
 a b      a b
k 0  k 
Let a=b=1 ,
Let
n n
n
2    
k 0  k 
a  p,b 1 p,
n  n k
n
nk
1     p 1 p    p  k 
n
k 0  k 
k 0
Ex.2.37. A binary channel with BER  103 .
3 bits are transmitted.
majority decoding is used. What is prob. of error?
 3
 3
2
P  k 2    .001 .999     .0013  3106
 2
 3
tch-prob
38
Multinomial Probability Law
P  B j   p j
Bj’s: partition of S; mutually exclusive
p1  p2  pM 1
n independent repetitions of experiment
B1
B2
.... ....
S
BM
k j number of times Bj occurs.
P[(k1, k2 ,..., kM )] 
n!
p1k1 p2k2 ... pM kM
k1 !k2 !...kM !
Ex.2.38. dart
3 areas
p  0.2,0.3,0.5
throw dart 9 times, P[3 on each area]=?
P  3,3,3  
9!
3 .3 3 .5 3
.2
     
3!3!3!
tch-prob
39
Geometric Probability Law
Repeat independent Bernoulli trials until the occurrence of the first success.
p  m   1 p ...1 p  p  1 p m1 p,
m1
m1,2,...
times


1
m 1 p
p
m

p
1

p
1
 

  
11 p  
m 1
m 1
P[ more than k trials are required before a success occurs ]

 p[m  k ]  p  1 p m11 p k
mk 1
P[The initial k trials are failures]
tch-prob
40
Sequence of Dependent Experiments
Ex.2.41. A sequential experiment involves repeatedly drawing a ball from one
of two urns, noting the number on the ball, and replacing the ball in its urn.
Urn 0 : 1 ball #1, 2 balls #0
Urn 1 : 5 balls #1, 1 ball #0
Initially, flip a coin. Thereafter, the urn used in the subexperiment corresponds
to the number selected in the previous subexperiment.
0
0
0
0
0
h
1
1
1
0
....
Urn 2
0
1
S
Urn 1
1
0
t
....
....
1
1
S
1
S
2
S
3
Trellis diagram
tch-prob
41
P S S   P S S  P S 
conditional prob.
 0 1
 1 0  0
PS S S PS S S  PS S 
 0 1 2   2 0 1  0 1
PS S S  PS S  PS 
 2 0 1  1 0   0 
PS S  PS S  PS 
Markov property
 2 1  1 0   0 
P  S0 S1 ,... Sn   P  Sn Sn1  P  Sn1 Sn2  ...P  S1 S0  P  S0 
....
Ex.2.42 calculate
with
P 0011  P 1 1 P 1 0  P 0 0  P 0 
1
2
1
P1 0
P 0 0
P 0  P[1]
3
3
2
1
5
P 1 1 
P 0 1
6
6
tch-prob
42
Problem 95 on Page 83
95 Consider a well-shuffled deck of cards consisting of 52 distinct cards, of
which four are aces and four are kings.
a. Pr[Obtaining an ace in the first draw]=
4 1

52 13
b. Draw a card from the deck and look at it.
Pr[Obtaining an ace in the second draw]=
A
3
51
A
4
51
If does not know the previous outcome
1
13
4
 513  12
13  51 
1
13
tch-prob
43
c. Suppose we draw 7 cards from the deck.
Pr[7 cards include 3 aces]=
Pr[7 cards include 2 kings]=
 4  48 
 3  4 
 

 52 
 7 


 4  48 
 2  5 
 

 52 
 7 


Pr[7 cards includes 3 aces and 2 kings]=
 4  4  44 
 3  2  2 
  

52


 7 


d. Suppose the entire deck of cards is distributed equally among four
players. Pr[each player gets an ace]=
 4  48  3  36  2  24 13 
 1  12  1  12  1  12 13  4!
       
 52  39  26 13 
 13  13  13 13 
    
4!
tch-prob
44
Birthday problem.
p =No two people in a group of n people will have a common birthday.
1 
2   n 1 

p   1
1

 365   1 365 
365


 

1
p
n  23
2
p  0.01
n 56
Lotto Problem.
Six numbers and one special number are picked from a pool of 42
numbers. …
1.
What is the probability of winning the grand prize?
2.
What is the probability of winning 2nd prize?
3.
What is the probability that number 37 is drawn next time?
tch-prob
45