Download Basic Concepts of Discrete Probability

Basic Concepts of Discrete
Probability
(Theory of Sets: Continuation)
1
Functions
• If X is a set and Y is a set, and there is a
sequence of well-specified operations for
assigning a well-defined object y  Y to every
element x  X , and by applying these
sequence of operations to every member of a
set X we obtain a set Y, then this sequence of
operations forms a rule, which is commonly
known as function.
2
Functions
• The set X is called the domain of the function,
and the set Y is called the range.
• Functions that have numerical values ere
called numerical functions.
• A numerical function establishing the
correspondence between the elements of a
set having a finite number of elements and a
set of natural numbers will be of our
particular interest:
n( X ) | X |
3
The number of elements in a set
• The number of elements in a set may or may
not be finite.
• If the elements of the set can be placed in
one-to-one correspondence with the set of
natural numbers {1, 2, 3,…}, we say that the
set has a denumerable or countable number
of elements; otherwise the set is
nondenumerable (e.g., a set of points on any
line or the set of points in [0,1]).
4
The number of elements in a set
• If a set is denumerable and n( X ) | X | is a
finite number, we say that the power of the
set X (the cardinality of the set X) is |X| (the
number of elements in the set X).
• If a set is equivalent to the set of points in
[0,1] that is, it is denumerable, we say that the
set has the power of continuum.
5
The number of elements in a set
• The following properties hold for finite sets::
• If A and B are two disjoint sets, then
n( A  B)  n( A)  n( B)
• For any finite sets A, B, and C:
n( A  B)  n( A)  n( B)  n( AB)
n( A  B)  n( A)  n( AB)
n( A)  n( A)  n(U )
n( A  B  C )  n  A  ( B  C )   n( A)  n( B  C )  n  AB  AC  


 n( A)  n( B)  n(C )  n( BC )   n( AB)  n( AC )  n( ABAC )  
ABC


 n( A)  n( B)  n(C )  n( BC )  n( AB)  n( AC )  n( ABC )
6
Basic Concepts of Discrete
Probability
Elements of the Probability Theory
7
Sample Space
• When “probability” is applied to something,
we usually mean an experiment with certain
outcomes.
• An outcome is any one of the possibilities that
may be expected from the experiment.
• The totality of all these outcomes forms a
universal set which is called the sample space.
8
Sample Space
• For example, if we checked occasionally the
number of people in this classroom on
Tuesday from 7pm to 9-45pm, we should
consider this an experiment having 15
possible outcomes {0,1,2,…,13,14} that form a
universal set.
• 0 – nobody is in the classroom, … 14 – all
students taking the Information Theory Class
and the instructor are in the classroom
9
Sample Space
• A sample space containing at most a
denumerable number of elements is called
discrete.
• A sample space containing a nondenumerable
number of elements is called continuous.
10
Sample Space
• A subset of a sample space containing any
number of elements (outcomes) is called an
event.
• Null event is an empty subset. It represents an
event that is impossible.
• An event containing all sample points is an
event that is certain to occur.
11
Probability Measure
• The probability measure is a specific type of
function which can be associated with sets.
• Since an experiment is defined so that to each
possible outcome of this experiment there
corresponds a point in the sample space, the
number of outcomes of the experiment is
assumed to be at most denumerable.
12
Probability Measure
• Let us consider an event of interest A as the
set of outcomes ak.
• Let a real function m(ak ) be the probability
measure of the outcome ak.
• The probability measure of an event is defined
as the sum of the probability measures
associated with all the outcomes ak of that
event. It is also referred to as the “additive
n
probability measure”: m{ A}  m(a )

k 1
k
13
Probability Measure
• The additive probability measure has the
following important property: the measure
associated with the union of two disjoint sets
is equal to the sum of their individual
measures.
14
Probability Measure
• Two events A and B are called disjoint if they
contain no outcome in common: two disjoint
events cannot happen simultaneously. The
probability measure has the following
assumed properties:
0  m ak 
m  A  B  m{ A}  m{B} if A and B are disjoint
m{ X }  0 if X  
m{ X }  1 if X  U
15
Probability Measure
• If A  U , B  U , C  U with an additive
probability measure, then the following
relations are valid:
m{ A}  m{B}

 if A  B
m{ A}  m{B}  m{B  A}
m{ A}  m{U  A}  m{U }  m{ A}  1  m{ A}


m{ A  B}  m ( A  AB)  B   m{ A}  m{ AB}  m{B} 


A B
 m{ A}  m{B}  m{ AB}
m{ A}  m{B}  m{ AB}
16
Probability Measure
• For three disjoint sets A  U , B  U , C  U :
m{ A  B  C}  m{ A}  m{B}  m{C}
• For three sets in general:
m{ A  B  C}  m{ A}  m{B}  m{C} 
 m{AB}  m{BC}  m{CA}  m{ABC}
17
Frequency of events
• Let us consider a specific event Xk among all
the possible events of the experiment under
consideration. If the basic experiment is
repeated N times among which the event Xk
has appeared n  X k  times, the ratio
n Xk  / N
is defined as the relative frequency of the
occurrence of the event Xk.
18
Frequency of events.
The Probability
• If N is increased indefinitely, that is if N  
then, intuitively speaking,
n Xk 
lim
 PX k 
N 
N
is the probability of the event Xk.
19
The Probability
• The classical definition of Laplace says that the
probability is the ratio of the number of
favorable events to the total number of
possible events.
• All events in this definition are considered to
be equally likely: e.g., throwing of a true die
by an honest person under prescribed
circumstances…
• …but not checking of the number of people in
the classroom.
20
The Probability
• The following properties are important:
0  n Xk   N
n Xk 
0
1
N
n Xk 
0  lim
1
N 
N
 0  PX k   1
21
The Probability
• To verify whether our “empiric” definition of
probability satisfies the properties of the
probability measure, we have to consider the
probability of two events.
• Let A and B be the events among the ones
resulting from the experiment. Let the
experiment be repeated n times.
22
The Probability
• Each observation can belong to only one of
the four following categories:
• 1) A has occurred, but not B  AB’; A  AB ' AB
• 2) B has occurred, but not A  BA’; B  BA ' AB
• 3) Both A and B have occurred  AB;
A  B  AB ' AB  BA '
• 4) Neither A nor B has occurred  A’B’
23
The Probability
• If the number of events of each category from
the mentioned four ones is denoted by n1 , n2 , n3 , n4
then n1  n2  n3  n4  n
•
•
n1  n3
f  A 
n
relative frequency of A
independent of B
n2  n3
f  A 
n
relative frequency of B
independent of A
24
The Probability
•
n1  n2  n3
f  A  B 
n
relative frequency of either
A, B or both
n3
• f  AB  n relative frequency of A and B
occurring together
n3
• f  A | B  n2  n3 relative frequency of A under
condition that B has occurred
n3
• f B | A  n  n relative frequency of B under
1
3
condition that A has occurred
25
The Probability
• When the number of experiments tends to
infinity, these relations lead to:
P  A  B  P{ A}  P{B}  P{ AB}
P{ AB}  P{ A}P{B | A}
P{ AB}  P{B}P{ A | B}
• If A and B are mutually exclusive events, then
P{AB}=0; P{A+B}=P{A}+P{B}
26
The Probability
• Thus, we have proved that
0  P  A  1
P  A  B  P{ A}  P{B} for mutually exclusive A and B
P{ X }  0 if, and only if, X  
P{ X }  1 if, and only if, X  U
• Let us compare the latter with the properties
of the probability measure:
0  m ak 
m  A  B  m{ A}  m{B} if A and B are disjoint
m{ X }  0 if X  
m{ X }  1 if X  U
27
The Probability
• We have just shown that our empirical
definitions of the frequency and probability,
respectively, satisfy all the properties of the
probability measure.
28
Theorem of addition
• For two events A and B of the sample space:
A  ( B  AB)  A  B
• The additive property of the probability
measure suggests that
m{ A  B}  m{ A}  m{B}  m{ AB}
P{ A  B}  P{ A}  P{B}  P{ AB}  P{ A}  P{B}
• If A and B are mutually exclusive events, then
P{ AB}  P{}  0
P  A  B  P{ A}  P{B}
29
Theorem of addition
• For two opposite events A and A’: A+A’=U;
AA’=0, then
P  A  A '  P{ A}  P{ A '}  P{U }  1
P{ A '}  1  P{ A}
30
Theorem of addition
• For the three events A, B, and C:
P  A  B  C  P  A  B  P{C}  P ( A  B)C 
 P{ A}  P{B}  P{C}  P{ AB}  P{BC}  P{CA}  P{ ABC}
• In general, for a number of events A1 , A2 ,..., An
P  A1  A2  ...  An   P  A1  P  A2   ...  P  An  
 P  A1 A2   P  A1 A3   ...  P  An 1 An   P  A1 A2 A3   P  A1 A2 A4  
...  P  An 2 An 1 An   ...  (1) n 1 P  A1 A2 ... An 
31
Theorem of addition
• If the events are mutually exclusive, then
P  A1  A2  ...  An   P  A1  P  A2   ...  P  An 
• It is also clear now that
P  A1  A2  ...  An   P  A1  P  A2   ...  P  An 
32
Conditional Probability
• Let A and B be two events. The conditional
probability of event A based on the hypothesis
that event B has occurred is defined by the
following relation:
P  AB
P  A | B 
if P  B  0
P  B
33
Conditional Probability
•
•
•
•
•
Returning to the example of two events
1) A has occurred, but not B  AB’ n1;
2) B has occurred, but not A  BA’ n2;
3) Both A and B have occurred  AB n3;
4) Neither A nor B has occurred  A’B’ n4:
f  AB
n3
f  A | B 

;
n2  n3
f  B
f  AB
n3
f  B | A 

n1  n3
f  A
34
Conditional Probability
• The two events A and B are called mutually
independent if P{A|B}=P{A} and P{B|A}=P{B}.
• For mutually independent events
P{AB}=P{A}P{B}.
35
Theorem of multiplication
• The multiplication rule for the case of two
events A and B can be obtained through the
definition of the conditional probability:
P{AB}=P{A}P{B|A}=P{B}P{A|B}.
• For three events A,B, and C:
P{ABC}=P{AB}P{C|AB}=P{A}P{B|A}P{C|AB}
36
Theorem of multiplication
• For n events:
P  A1 A2 ... An  
 P{ A1}P{ A2 | A1}P  A3 | A1 A2  ...P{ An | A1 A2 ... An 1}
• For a countable infinite number of the
mutually independent events
P  A1 A2 ...An   P{A1}P{A2}...P{An }
37