Download Probability theory refresher

CPSC 531: Basic Probability Concepts
Instructor: Anirban Mahanti
Office: ICT 745
Email: [email protected]
References:
1.
“Introduction to Probability Models” by Sheldon Ross,
Academic Press, Eight Edition, 2003.
2.
“Probability and Statistics” by M. DeGroot and M.
Schervish, Third edition, Addison Wesley, 2002.
3.
“Probability and Random Processes for Electrical and
Computer Engineers” by John Gubner, Cambridge Press,
First Edition, 2006.
Basic Probability Concepts
1
Outline
ˆ Experiments, sample space, and events
ˆ Review of set theory
ˆ Probability: definition, property,
interpretation
ˆ Conditional probability
ˆ Independent events
ˆ Law of total probability and Bayes’
ˆ Counting Methods
Basic Probability Concepts
2
Why Study Probability Theory?
ˆ Probability theory is useful in understanding,
studying, and analyzing complex real world
systems
ˆ Probability theory can be used to model and
develop complex real world systems
ˆ A good understanding of probability theory is
needed to develop simulations

E.g., How would you model network traffic?
• Should we use Markovian models?
• What is the impact of “self similarity” on design and
analysis of communication networks
Basic Probability Concepts
3
1
Experiment
ˆ Experiment – in probability theory refers to a
process whose outcome is not known in advance
with certainty
E.g., Suppose a coin is tossed 10 times. How many
times will we get “heads”?
 Experience tells us that, if the coin is fair, we will
see on average 5 “heads”; we can arrive at this
result by performing the experiment many times
and noting down the observations
 Instead of performing the experiments, we can use
probability theory to develop a model of any system
that yields uncertain measurements

Basic Probability Concepts
4
Sample space, Outcome, Event
ˆ Sample space (S) – is the collection of all
possible outcomes of a system or experiment.
E.g., Rolling a six-sided dice can be modeled with
the sample space S = {1, 2, 3, 4, 5, 6}
 E.g., Model of the number of packets queued at a
router that can buffer up to 50 packets would use
the sample space S = {0, 1, 2, …, 50}

ˆ Outcome – is element in the sample space
 E.g., 2, 4, and 5 are outcomes in the rolling dice
example
ˆ Event – is a subset of the sample space
 E.g., A is an event that an even number is rolled:
A = {2, 4, 6}
Basic Probability Concepts
5
Review of Set Theory
Basic Probability Concepts
6
2
Countable vs. Uncountable Sets
ˆ Some sets contain only finite number of
elements, while others contain infinitely many
elements. An infinite set can be classified as:

Countable - if there is a one-to-one correspondence
between elements of the set and the set of natural
numbers
• E.g., the set of numbers from the roll of a dice
• E.g., the set of all odd numbers
• E.g., the set of all prime numbers

Uncountable – converse of countable; e.g., set of
real numbers, numbers in the interval [0, 1]
• E.g., set of possible temperatures on Jan 1 in a city
Basic Probability Concepts
7
Set Operations (1)
ˆ Consider a set of points: S
ˆ Let A and B be two collection of points in S
ˆ Every point in A is also in B, then A ⊂ B
 If A ⊂ B and B ⊂ A, then A = B
ˆ Empty set: Ø is the “null” set
ˆ Complement: AC is the complement of set A
and contains all outcomes in the sample space
S that do not belong to A
Basic Probability Concepts
8
Set Operations (2)
ˆ Union: A ∪ B = a set that consists of points
that belong either to A or to B, or to both

A ∪ B := {w Є S: w Є A or w Є B}
ˆ Intersection: A ∩ B = a set that consists of
points that belongs to both A and B

A ∩ B := {w Є S: w Є A and w Є B}
ˆ Two sets are said to be disjoint or mutually
exclusive if A ∩ B = Ø
ˆ Two sets are said to be mutually exclusive and
exhaustive if A ∪ B = S and A ∩ B = Ø
Basic Probability Concepts
9
3
Set Identities (1)
ˆ Commutative laws
A ∪B =B ∪A
A ∩B =B ∩A
ˆ Distributive laws
 A ∪ (B ∩ C) = (A ∪ B) ∩ (A ∪ C)
 A ∩ (B ∪ C) = (A ∩ B) ∪ (A ∩ C)
ˆ Associative laws
 (A ∪ B) ∪ C = A ∪ (B ∪ C) = (A ∪ B) ∪ C
 (A ∩ B) ∩ C = A ∩ (B ∩ C) = (A ∩ B) ∩ C
Basic Probability Concepts
10
Basic Probability Concepts
11
Set Identities (2)
ˆ Identity laws
A ∪Ø = A
A ∩U =A
A ∪U =U
A ∩Ø = Ø
ˆ De Morgan’s laws
 (A ∪ B)C = AC ∩ BC
 (A ∩ B)C = AC ∪ BC
Definitions of Probability
Basic Probability Concepts
12
4
Classical Definition
ˆ In the classical approach, the probability of an
event P(A) is assigned a priori without
performing any experiments.
ˆ The probability of an event A is the ratio of
the number of outcomes NA favorable to event
A to the total number N of possible outcomes
of the experiment.
P (A) =
NA
N
Basic Probability Concepts
13
Relative Frequency Definition
ˆ Suppose n trials of an experiment are
undertaken. In these trials, an event A occurs
nA times. The probability of event A, P(A), can
be defined as follows:
P (A) = lim
n→∞
nA
n
ˆ Note that the probability is assigned after (a
posterior) the experiments.
Basic Probability Concepts
14
Axiomatic Definition
ˆ A probability measure, or simply a probability,
on a sample space S is a specification of
numbers P(.) that satisfies the following
axioms:
1)
2)
3)
0 ≤ P(A) ≤ 1 for every event A; probability of an
event A is a number between 0 and 1.
P(S) = 1; the probability of an outcome being a
sample point in the sample space is 1.
For every infinite sequence of mutually exclusive
events A1, A2, …
⎛ ∞
⎞
∞
P ⎜⎜ U Ai ⎟⎟ = ∑ P(A i )
⎝ i =1 ⎠ i =1
Basic Probability Concepts
15
5
Probability Models (1)
ˆ Probability theory helps develop mathematical
models of complicated problems

Example: Develop a mathematical model of rolling a
fair die to determine “probability” of an even face.
Let S = {1, 2, 3, 4, 5, 6} be the sample space.
Let Ei = {i}, i = 1, 2, …, 6 be the event that die rolled
the face i.
Let E = {2, 4, 6} model the die rolling an even face.
Here the classical definition of probability is used.
P(E) = |E|/|S| = 1/2
Basic Probability Concepts
16
Probability Models (2)

Some observations about the probability model:
•
•
•
•

P(Ø) = |Ø|/|S| = 0.
P(A) ≥ 0 for every event A.
If A and B are disjoint events, then P(A ∪ B) = P(A) + P(B)
When the die is rolled, something happens; this is represented
by the fact that P(S) = 1.
Our example considered the countable finite case. What
about countable infinite outcomes?
• Let S = {1, 2, …}. For w Є S, let p(w) be non-negative and
∞
∑ p( w) = 1.
w =1
For any A ⊂ S, P(A) = ∑ p(w).
Basic Probability Concepts
17
Probability Models (3)

In many cases, the number of possible outcomes are
uncountable. E.g., the duration of Web browsing sessions, the
duration of a phone call, the heights of adults in a city etc.
We can define P as follows:
P (A) =
Z
f (w)dw,
A
A⊂S
for some non-negative function f such that:
∫ f ( w)dw = 1.
S
Basic Probability Concepts
18
6
Properties of Probability
ˆ (Complement) For any event A, P(AC) = 1 – P(A)
ˆ (Impossible Set) P(Ø) = 0
ˆ (Monotonicity rule) If A ⊂ B, P(A) ≤ P(B)
ˆ (Inclusion-Exclusion rule) Given two events A
and B, P(A U B) = P(A) + P(B) – P(A ∩ B).
ˆ Other rules can be derived. The inclusion-
exclusion rule can be extended as follows:
Suppose A, B, and C are events in S. Then
P (A∪B∪C) = P (A)+P (B)+P (C)−P (A∩B)−P (A∩C)−P (B∩C)+P (A∩B∩C)
Basic Probability Concepts
19
Example (1)
ˆ Select one ball from a box containing white
(W), red (R), blue (B), and green (G) balls.
Suppose that P(R) = 0.1 and P(B) = 0.5. What is
the probability of selecting a white or a green
ball?
ˆ Solution:
WURUBUG=S
From axiom 3, we have P(S) = P(W) + P(R) + P(B) + P(G)
Also, P(WUG) = P(W)+P(G) = 1 – P(R) – P(B) = 0.4
Basic Probability Concepts
20
Example (2)
ˆ Rolling a fair dice: P(1) = P(2) = P(3) = P(4) =
P(5) = P(6) = 1/6, where P(i) is the probability
of rolling a face with i dots.
ˆ Question:


P({1, 3}) = ?
P({2, 4, 6}) = ?
Basic Probability Concepts
21
7
Conditional Probability
Basic Probability Concepts
22
Conditional Probability (1)
ˆ The conditional probability of event A given that
event B has occurred is
P( A | B) =

P( A ∩ B)
P( B)
The conditional probability P(A|B) is undefined if P(B)=0
ˆ The frequency interpretation: If an experiment is
repeated a large number of times, then event B occurs
approximately P(B) fraction of the time and the event
A and B both occur approximately P(A ∩ B) portion of
time. Therefore, among the repetitions in which B
occurs, the proportion of events in which A also occurs
is P(A ∩ B) /P(B). P(A ∩ B) also denoted as P(AB).
Basic Probability Concepts
23
Conditional Probability (2)
ˆ Example: Two balls are selected at random, without
replacement, from a bag containing r red balls and b
blue balls. Determine probability p that the first
selected ball is red and the second selected ball is
blue.
Solution
A = event that first ball is red; B = event that second
ball is blue
P(A) = prob. first ball is red = r/(r+b)
P(B|A) = prob. blue in second draw = b/(r+b-1)
P(B|A) = P(AB)/P(A)
P(AB) = (r.b)/((r+b)(r+b-1))
Basic Probability Concepts
24
8
Conditional Probability (3)
ˆ P(AB) = P(A) P(A|B); extended, as shown next
ˆ Suppose that A1, A2, A3, …, An are events such that
P(B) > 0 and P(A1A2A3 …An-1) > 0. Then,
P(A1A2A3…An) = P(A1)P(A2|A1) P(A3|A1A2)… P(An|A1A2A3 …An-1)
ˆ Example: Draw 4 balls, one by one, without
replacement, from a box containing r red balls and b
blue balls. Probability of sequence red, red, blue, blue?
Solution:
Let Rj and Bj denote event that a red and blue ball is
drawn, respectively.
P(R1R2B3B4) = P(R1)P(R2|R1)P(B3|R1R2) P(B4|R1R2B3)
complete the calculations on your own …
Basic Probability Concepts
25
Independent Events (1)
ˆ Independent events are those that don’t have
any effect on each other. That is, knowing one
of them occurs does not provide any
information about the occurrence of the other
event. Mathematically, A and B are
independent if P(A|B) = P(A) and P(B|A) = P(B)
ˆ From conditional probability definition, we
have P(AB) = P(A|B)P(B). Therefore, if A and B
are independent events, P(AB) = P(A)P(B)
Basic Probability Concepts
26
Independent Events (2)
ˆ Example: Determine the probability pn that
exactly n tosses are required for a head to
appear for the first time.
Solution
We need to determine the probability of a
sequence of (n-1) tails followed by a head. The
probability of obtaining a heads or tails (for a
fair coin) is ½. Therefore, pn = (½)n.
Basic Probability Concepts
27
9
Law of Total Probability and
Bayes’ Rule
Basic Probability Concepts
28
Law of Total Probability
ˆ Let B1, B2, B3, …, Bk be mutually disjoint and collectively
exhaustive events from the sample space S. Then, for
any event A in S, we have
k
P( A) = ∑ P( B j ) P( A | B j )
Explanation
A = (B1A)U (B2A)U… (BkA).
The (BjA)’s are disjoint
events. Therefore, using
laws of conditional
probability we get:
k
k
j =1
j =1
P ( A) = ∑ P ( B j A) = ∑ P( B j ) P ( A | B j ),
j =1
B1
B2
A
B3
B4
if P ( B j ) > 0 for j = 1,..., k
Basic Probability Concepts
29
Bayes’ Theorem
ˆ Partition: The events B1, B2, B3, …, Bk form a
partition of a set S if they are mutually
disjoint and Uik=1 Bi = S
ˆ Bayes’ Theorem: Suppose that B1, B2, B3, …, Bk
form a partition of sample space S such that
P(Bj)> 0 for j = 1, …, k. Let A be an event in S
such that P(A)>0. Then, for i = 1, …, k,
P( Bi | A) =
P( Bi ) P( A | Bi )
∑ kj =1 P( B j ) P( A | B j )
Basic Probability Concepts
30
10
Counting Methods
Basic Probability Concepts
31
Types of Counting Problems
ˆ Two types of counting problems:
 Selection problems – select k balls from an urn
containing n distinguishable balls
 Allocation problems – allocate k indistinguishable
balls to a set of n distinguishable urns
ˆ Consider the selection problem
 How do we replace selected balls?
• With replacement – put ball back into urn
• Without replacement

How to count events?
• Permutation – order of selection important
• Combination – order of selection unimportant
Basic Probability Concepts
32
Permutations (1)
ˆ The number of ordered ways k items can be
selected without replacement from a pool of n
items is:
Pn,k = n x (n-1) x … X (n-k+1) = n!/(n-k)!
ˆ The number of ordered ways k items can be
selected with replacement from a pool of n
items is:
Prn,k = n x n x … (k times) = nk
Basic Probability Concepts
33
11
Permutations (2)
ˆ When not all items are ordered: E.g., a club has
25 members and 2 officers (a president and a
secretary). These officers are to be chosen
from the members of the club. How many ways
can the officers be selected?

25 ways to select the first officer and 24 ways to
select the second officer. Answer = 25x24 = 600
Basic Probability Concepts
34
Permutations (3)
ˆ What is the probability that at least two
people share the same birthday? You may
assume the following:
365 possible birthdays (ignore leap years)
Our group consists of 2 ≤ k ≤ 365 people who are
unrelated (no twins)
 Each birthday is equally likely (birth rate
independent of the time of year)


Basic Probability Concepts
35
Combinations (1)
ˆ The number of combinations possible when
selecting k items from a set of n items
without replacement is denoted by Cn,k.
ˆ Note that one way to find Pn,k is to compute
Cn,k (i.e., unordered selection of k items) and
multiply by the number of ways of ordering k
items (i.e., k!).

Therefore, Pn,k = Cn,k x k!
Cn , k =
Pn, k
k!
=
n!
k!(n − k )!
Basic Probability Concepts
36
12
Combinations (2)
ˆ The number of combinations possible when
selecting k items from a set of n items with
replacement is denoted by Crn,k.
Crn,k = Cn+k-1,k.
Basic Probability Concepts
37
Combinations (3)
ˆ Example: A fair coin is tossed 10 times. Determine
the probability, p, of obtaining exactly 3 heads.
Solution



Each experiment has two possible outcome, heads or tails.
Since the coins are fair, we can assume that each outcome is
equally likely. Therefore, the total number of outcomes in
this experiment is = 210
The number of different arrangements possible with 3
heads and 7 tails is = C10,3
Therefore,
p=
C10,3
210
= 0.1172
Basic Probability Concepts
38
Summary
ˆ A review of basic probability and statistics
was presented. You probably enjoyed it!
ˆ Next, we will talk about discrete and
continuous random variables.
Basic Probability Concepts
39
13