Download Probability Theory Recap - Part 1

Probability Theory
Part 1: Basic Concepts
Sample Space - Events
 Sample Point
 The outcome of a random experiment
 Sample Space S
 The set of all possible outcomes
 Discrete and Continuous
 Events
 A set of outcomes, thus a subset of S
 Certain, Impossible and Elementary
Set Operations
 Union A  B
 Intersection A  B
 Complement AC
S
A B
 Properties
 Commutation
A B  B  A
 Associativity
A   B  C    A  B  C
 Distribution
AC
A   B  C    A  B   A  C 
 De Morgan’s Rule
 A  B
C
 AC  B C
A B
Axioms and Corollaries




Axioms
0  P  A
PS   1
If A  B  
P  A  B  P  A  P  B
 If A1, A2, … are
pairwise exclusive
  
P  Ak    P  Ak 
 k 1  k 1
 Corollaries
C
 P  A   1  P  A
 P  A  1
 P   0
 P  A  B 
P  A  P  B   P  A  B 
Computing Probabilities Using
Counting Methods
 Sampling With Replacement and Ordering
k
 n
 Sampling Without Replacement and With Ordering

n  n  1 ...  n  k  1
 Permutations of n Distinct Objects
 k!
 Sampling Without Replacement and Ordering
 n
 n 
n!
 

 k   n  k  k ! n  k  !
 Sampling With Replacement and Without Ordering
  n 1  k 


k
 n 1 k 


n

1
 

Conditional Probability
 Conditional Probability of
event A given that event
B has occurred
P  A | B 
A B
S
P  A  B
P  B
 If B1, B2,…,Bn a
partition of S, then
B1
P  A  P  A | B1  P  B1   ... 
B2
P  A | B j  P  B j 
(Law of Total Probability)
A B
AC
A
B3
Bayes’ Rule
 If B1, …, Bn a partition
of S then
P  A  B j 
P  B j | A 
P  A

P  A | B j  P  B j 
n
 P A | B  PB 
k 1
k
Example
Which input is more probable
if the output is 1? A priori,
both
input
symbols
are
equally likely.
input
0
k
likelihood  prior
posterior 
evidence
output 0
1-ε
ε
1-p
p
1
0
1
1
ε
1-ε
Event Independence
 Events A and B are
independent if
P  A  B  P  A P  B
 If two events have nonzero probability and are
mutually exclusive, then
they cannot be
independent
A
B
1
1
½
1
C
½
1
½
1
P  A  B   P  A P  B 
P  B  C   P  B  P C 
½
P  A  C   P  A P C 
½
1
P  A  B  C   P  
 P  A P  B  P  C 
Sequential Experiments

Sequences of Independent
Experiments



E1, E2, …, Ej experiments
A1, A2, …, Aj respective
events
Independent if

Bernoulli Trials


P  A1  A2  ...  An  
Test whether an event A
occurs (success – failure)
What is the probability of k
successes in n independent
repetitions of a Bernoulli
trial?
n
nk
pn  k     p k 1  p 
k 
n
n!

 
 k  k ! n  k  !
P  A1  P  A2  ...P  An 

Transmission over a
channel with ε = 10-3 and
with 3-bit majority vote