Download STAT 315: LECTURE 2 CHAPTER 2: PROBABILITY 1. Basic

STAT 315: LECTURE 2
CHAPTER 2: PROBABILITY
TROY BUTLER
1. Basic concepts, definitions, and notation
Some basic definitions:
• An experiment is an action or process whose outcome is subject to uncertainty.
• The sample space of an experiment, denoted by Ω or S, is the collection of all possible outcomes.
• An event is any (measurable) subset of the sample space. An event is simple if it consists of exactly
one outcome and compound if it consists of more than one outcome. We say an event occurs if the
outcome of the experiment falls in the subset defining the event.
• The complement of an event A, denoted as A0 (or Ac ), is the set of all outcomes in the sample
space that are not in A.
• The intersection of two events A and B, denoted as A ∩ B and read “A and B”, is the event
consisting of all outcomes that are in both A and B.
• The union of two events A and B, denoted as A ∪ B and read “A or B”, is the event consisting of
all outcomes that are either in A or in B or in both events.
• An null event is an event consisting of no outcomes, denoted by ∅.
• When A ∩ B = ∅, A and B are said to be mutually exclusive or disjoint events.
• The above definition can be extended to a collection of n events. Let A1 , A2 , ..., An be n events. We
say they are all mutually exclusive if, for every i 6= j, Ai ∩ Aj = ∅.
Venn diagrams are useful pictorial representations that are helpful in determining what some of these
definitions mean and in helping to prove some of the useful results. Some basic but very useful results for
events A, B, and C in sample space S:
• Commutativity: A ∪ B = B ∪ A, A ∩ B = B ∩ A.
• Associativity: A ∪ (B ∪ C) = (A ∪ B) ∪ C, and A ∩ (B ∩ C) = (A ∩ B) ∩ C.
• Distributive Laws: A ∩ (B ∪ C) = (A ∩ B) ∪ (A ∩ C), and A ∪ (B ∩ C) = (A ∪ B) ∩ (A ∪ C).
• DeMorgan’s Laws: (A ∪ B)0 = A0 ∩ B 0 , and (A ∩ B)0 = A0 ∪ B 0 .
• A ∩ ∅ = ∅.
1
2
TROY BUTLER
2. Probability axioms and interpretation
Given an experiment and a sample space S, the objective of probability is to assign to each event A a
number P(A), called the probability of the event A, which will give a precise measure of the chance that A
will occur. There are three axioms for the probability measure P .
AXIOM 1: For any event A, P (A) ≥ 0.
AXIOM 2: P (S) = 1
AXIOM 3: If A1 , A2 , ..., be an set of mutually exclusive events, then
P (A1 ∪ A2 ∪ A3 ∪ ...) =
∞
X
P (Ai ).
i=1
With the above axioms it is relatively straightforward to prove the following
Theorem 1. For any two events A and B in sample space S we have:
• P (A0 ) = 1 − P (A),
• P (∅) = 0, and
• P (A ∪ B) = P (A) + P (B) − P (A ∩ B).
What is the proper interpretation of probability, or, in other words, what do we mean when we say that the
probability of some event is a particular number? Consider an experiment that can be repeatedly performed
in an identical and independent fashion, and let A be an event consisting of a fixed set of outcomes of the
experiment. Suppose the experiment is performed n times, and let n(A) denote the number of replications
on which A does occur. The ratio n(A)/n is the relative frequency. As n gets arbitrarily large, the relative
frequency approaches a limiting value we call the limiting relative frequency of event A. The objective
interpretation of probability identifies this limiting relative frequency with P (A).
3. Counting techniques
When the various outcomes of an experiment are equally likely (the sample probability is assigned to
each simple event), the task of computing probabilities reduces to counting. Letting N denote the number
of outcomes in a sample space and N (A) represent the number of outcomes in an event contained in event
A, then P (A) =
N (A)
N .
3.1. Product rule. Let there be k operations, the ith operation can be completed in ni ways, i = 1, ..., k.
Qk
The total number of ways the process can be completed is i=1 ni .
3.2. Permutations. A permutation is an ordering of a collection of objects. Any ordered sequence of
k objects taken from a set of n distinct objects S is called a k-permutation of set S and the number of
k-permutations of set S is denoted by Pk,n .
STAT 315: LECTURE 2
CHAPTER 2: PROBABILITY
3
Theorem 2. Given n different objects, there are n! (read “n factorial”) permutations of the n objects, where
n! = n(n − 1)(n − 2)...(3)(2)(1),
and we define 0! = 1.
Theorem 3. The number of permutations of k objects selected from n objects is:
n!
(n − k)!
Pk,n =
Theorem 4. If you have n objects and wish to divide them into j groups of size k1 , k2 , ..., kj where
Pj
i=1
ki =
n, then there are
n!
Qj
i=1
ki !
different collections.
3.3. Combinations. An unordered subset of n objects is called a combination. Any unordered sequence
of k objects taken from a set of n distinct objects
 Sis called a k-combination of set S and the number of
n
.
k-combinations of set S is denoted by Ck,n or 
k
The number of combinations of k objects chosen from n objects is given by


n
n!

=
.
k!(n
− k)!
k
4. Conditional probability, LTP and Bayes’ Theorem
For any two events A and B where P (B) > 0, the conditional probability of A “given” B is given by
P (A|B) :=
P (A ∩ B)
.
P (B)
The multiplication rule follows directly from the definition of conditional probability. Just multiply
both sides by P (B) to get
P (A ∩ B) = P (A|B)P (B).
The Law of Total Probability says that, for any set of mutually exclusive and exhaustive events
A1 , A2 , ......, An (i.e. a partition of S) and any event B, we have that
P (B) =
n
X
P (Ai ∩ B) =
i=1
The term exhaustive means that ∪ni=1 Ai = S.
n
X
i=1
P (B|Ai )P (Ai ).
4
TROY BUTLER
Theorem 5 (Bayes’ Theorem). Let A1 , A2 , ..., An be n mutually exclusive and exhaustive events with prior
probabilities P (Ai ) for i ∈ 1, 2, ..., n. Then for any event B with P (B) > 0, the posterior probability for
event Aj , P (Aj |B), is defined as
P (B|Aj )P (Aj )
P (Aj |B) = Pn
.
i=1 P (B|Ai )P (Ai )
Note the right hand side of the definition simplifies to
P (Aj |B) =
P (Aj ∩ B)
.
P (B)
Some details about the prior and posterior:
• The prior is a probability that is know, or theorized before, or prior, to running an experiment.
• The posterior probability is the probability of the event after, or post, experiment.
5. Independence
Two events A and B, where P (A) 6= 0 and P (B) 6= 0 are said to be independent if P (A|B) = P (A) or
P (B|A) = P (B). By definition we say that A and B are independent if either P (A) = 0 or P (B) = 0.
Theorem 6. Let A and B be two events. A and B are independent if and only if
P (A ∩ B) = P (A)P (B).
In a more general case, A1 , A2 , ..., An are mutually independent if for every k (k = 2, 3, ..., n) and every
subset of indices i1 , i2 , ..., ik ,

P
k
\

Aij  =
j=1
k
Y
P (Aij ).
j=1
Theorem 7. Let A and B be any two independent events. Then the pairs (1) A, B 0 , (2) A0 , B, and (3)
A0 , B 0 are also independent.
6. Exercises to do in class
Chapter 2 exercises: 24, 26, 38, 56, 58, 80, 100