Download Conditional Probability

CPSC 531:Probability & Statistics:
Review
Instructor: Anirban Mahanti
Office: ICT 745
Email: [email protected]
Class Location: TRB 101
Lectures: TR 15:30 – 16:45 hours
Class web page:
http://pages.cpsc.ucalgary.ca/~mahanti/teaching/F05/CPSC531
Notes derived from “Probability and Statistics” by
M. DeGroot and M. Schervish, Third edition,
Addison Wesley, 2002.
CPSC 531: Probability Review
1
Outline
 Experiments, sample space, and events
 Review of set theory
 Probability: definition, property,
interpretation
 Finite sample space
 Counting methods
 Conditional probability
 Independent events
 Law of total probability
 Bayes’ Theorem
CPSC 531: Probability Review
2
Experiments, sample space, and events
 Experiment – in probability theory refers to a
process whose outcome is not known in advance
with certainty

E.g., An experiment consists of tossing a coin 10
times. What is the probability that at least four
heads are obtained in a row?
 Sample space (S) – is the collection of all
possible outcomes of an experiment. Each
outcome is a sample point.

E.g., Rolling a six-sided dice: S = {1, 2, 3, 4, 5, 6}
 Event – is a subset of the sample space
 E.g., A is an event that an even number is rolled:
A = {2, 4, 6}
CPSC 531: Probability Review
3
Set Theory (1)
 Some sets contain only finite number of
elements, while others contain infinitely many
elements. An infinite set can be classified as:
Countable - if there is a one-to-one correspondence
between elements of the set and the set of natural
numbers
 Uncountable – converse of countable; e.g., set of
real numbers, numbers in the interval [0, 1]

 Empty set: Ø is the “null” set
 Union: A  B = is the event containing all
outcomes that are in A or B, or both
CPSC 531: Probability Review
4
Set Theory (2)
 Intersection: A  B = event contained in both
A and B
 Complement: AC is the complement of event A
and contains all outcomes in the sample space
S that do not belong to A
 Idempotent laws
AA=A
A  A = A

 Associative laws
(A  B)  C = A  (B  C) = (A  B)  C
 (A  B)  C = A  (B  C) = (A  B)  C

CPSC 531: Probability Review
5
Set Theory (3)
 Commutative laws
AB=BA
A  B = B  A

 Distributive laws
A  (B  C) = (A  B)  (A  C)
 A  (B  C) = (A  B)  (A  C)

 Identity laws
AØ=A
A  U= A
A  U = U
A  Ø= Ø

CPSC 531: Probability Review
6
Probability - definition
 A probability distribution, or simply a
probability, on a sample space S is a
specification of numbers P(.) that satisfies
the following axioms:
1)
2)
3)
For every event A, P(A) ≥ 0
P(S) = 1
For every infinite sequence of mutually exclusive
events A1, A2, …
  



P  Ai    P(Ai )
 i 1  i 1
CPSC 531: Probability Review
7
Properties of Probability
 For any event A, P(AC) = 1 – P(A)
 Since Ø = SC, P(Ø) = 0
 If A  B, P(A) ≤ P(B)
 For every event A, 0 ≤ P(A) ≤ 1
 Home work: Recall Venn diagrams … they can
help in proving the above properties.
CPSC 531: Probability Review
8
Example (1)
 Select one ball from a box containing white
(W), red (R), blue (B), and green (G) balls.
Suppose that P(R) = 0.1 and P(B) = 0.5. What is
the probability of selecting a white or a green
ball?
 Solution:
WURUBUG=S
From axiom 3, we have P(S) = P(W) + P(R) + P(B) + P(G)
Also, P(WUG) = P(W)+P(G) = 1 – P(R) – P(B) = 0.4
CPSC 531: Probability Review
9
Example (2)
 Rolling a fair dice: P(1) = P(2) = P(3) = P(4) =
P(5) = P(6) = 1/6, where P(i) is the probability
of rolling a face with i dots.
 Question:
P({1, 3}) = ?
 P({2, 4, 6}) = ?

CPSC 531: Probability Review
10
Interpretations of probability
 Classical view: Probability measures the long-
term likelihood of outcomes. This is, if an
experiment is repeated “many” times, event A
should occur roughly P(A) fraction of the time.

Can “many” be quantified?
 Bayesian view: Probability represents a
personal estimate of how likely an event is
going to occur.
 Note: the theory of probability does not
depend on the interpretation. Good!
CPSC 531: Probability Review
11
Finite Sample Spaces
 A common assumption for a finite sample space
S containing n outcomes s1, s2, … sn is that each
outcome is equally likely.
 If an event A in this sample space contains
exactly m outcomes, then P(A) = m/n
 Example: Toss three fair coins simultaneously.
What is the probability of obtaining exactly
two heads?
S = {(HHH), (HHT), (HTH), (THH), (TTH),
(HTT), (THT), (TTT)}
P(2 H’s) = 3/8
CPSC 531: Probability Review
12
Counting Methods
 Multiplication rule
 Permutations
 Combinations
CPSC 531: Probability Review
13
Multiplication rule
 If an experiment has k “parts” (k ≥ 2), and the
ith part has ni possible outcomes, then the total
number of possible outcomes is n1 x n2 … x nn
 Example: 6 coins are tossed
6 parts to the experiments: toss 1, 2, …, 6
 2 possible outcomes per part (i.e., per toss)
 Possible outcomes = 26 = 64

CPSC 531: Probability Review
14
Permutations (1)
 Consider an experiment that involves sampling
without replacement. E.g., arrange 5 different
books on a shelf. How many ways can the books
be arranged?
 Think of the shelf as having 5 slots
 5 choices for filling the first slot
 4 choices for filling the second slot
 3 choices for filling the third slot
 2 choices for filling the fourth slot ….
 So, we have 5x4x3x2x1 = 120 possibilities
CPSC 531: Probability Review
15
Permutations (2)
 Given n items, the number of permutations (or
arrangements) of all n items equals n!
 n!
= n x (n-1) x (n-2) x … x 2 x 1
 Definition for 0! - It equals 1 since there is only 1
way to arrange zero items.
 When not all items are ordered: E.g., a club has
25 members and 2 officers (a president and a
secretary). These officers are to be chosen
from the members of the club. How many ways
can the officers be selected?

25 ways to select the first officer and 24 ways to
select the second officer. Answer = 25x24 = 600
CPSC 531: Probability Review
16
Permutations (3)
 Generalization: The number of ordered ways k
items can be selected from a pool of n items is:
Pn,k = n x (n-1) x … X (n-k+1) = n!/(n-k)!
 Home work problem: What is the probability
that at least two people share the same
birthday? You may assume the following:
365 possible birthdays (ignore leap years)
 Our group consists of 2 ≤ k ≤ 365 people who are
unrelated (no twins)
 Each birthday is equally likely (birth rate
independent of the time of year)

CPSC 531: Probability Review
17
Combinations (1)
 Consider choosing a subset containing k items
from a set of n distinct items. How many
different subsets are possible?

The arrangement of elements in the subset is
irrelevant. We treat each subset as an unit.
 Such unordered samples are called
combinations. The number of combinations
possible when selecting k items from a set of n
items is denoted by Cn,k.
 We will use the multiplication rule to find Cn,k.
CPSC 531: Probability Review
18
Combinations (2)
 Note that one way to find Pn,k is to compute Cn,k
(i.e., unordered selection of k items) and
multiply by the number of ways of ordering k
items (i.e., k!).
 Therefore, Pn,k = Cn,k x k!
 That is,
Cn, k
Pn, k
n!


k!
k!(n  k )!
CPSC 531: Probability Review
19
Combinations (3)
 Example: A fair coin is tossed 10 times. Determine
the probability, p, of obtaining exactly 3 heads.
Solution



Each experiment has two possible outcome, heads or tails.
Since the coins are fair, we can assume that each outcome is
equally likely. Therefore, the total number of outcomes in
this experiment is = 210
The number of different arrangements possible with 3
heads and 7 tails is = C10,3
Therefore,
p
C10,3
10
 0.1172
2
CPSC 531: Probability Review
20
Conditional Probability (1)
 The conditional probability of event A given that
event B has occurred is
P( A | B) 
P( A  B)
P( B)
 The conditional probability P(A|B) is undefined if
P(B)=0
 The frequency interpretation: If an experiment is
repeated a large number of times, then event B occurs
approximately P(B) fraction of the time and the event
A and B both occur approximately P(A  B) portion of
time. Therefore, among the repetitions in which B
occurs, the proportion of events in which A also occurs
is P(A  B) /P(B). P(A  B) also denoted as P(AB).
CPSC 531: Probability Review
21
Conditional Probability (2)
 Example: Two balls are selected at random, without
replacement, from a bag containing r red balls and b
blue balls. Determine probability p that the first
selected ball is red and the second selected ball is
blue.
Solution
A = event that first ball is red; B = event that second
ball is blue
P(A) = prob. first ball is red = r/(r+b)
P(B|A) = prob. blue in second draw = b/(r+b-1)
P(B|A) = P(AB)/P(A)
P(AB) = (r.b)/((r+b)(r+b-1))
CPSC 531: Probability Review
22
Conditional Probability (3)
 P(AB) = P(A) P(A|B); extended, as shown next
 Suppose that A1, A2, A3, …, An are events such that
P(B) > 0 and P(A1A2A3 …An-1) > 0. Then,
P(A1A2A3…An) = P(A1)P(A2|A1) P(A3|A1A2)… P(An|A1A2A3 …An-1)
 Example: Draw 4 balls, one by one, without
replacement, from a box containing r red balls and b
blue balls. Probability of sequence red, red, blue, blue?
Solution:
Let Rj and Bj denote event that a red and blue ball is
drawn, respectively.
P(R1R2B3B4) = P(R1)P(R2|R1)P(B3|R1R2) P(B4|R1R2B3)
complete the calculations on your own …
CPSC 531: Probability Review
23
Independent Events (1)
 Independent events are those that don’t have
any effect on each other. That is, knowing one
of them occurs does not provide any
information about the occurrence of the other
event. Mathematically, A and B are
independent if P(A|B) = P(A) and P(B|A) = P(B)
 From conditional probability definition, we
have P(AB) = P(A|B)P(B). Therefore, if A and B
are independent events, P(AB) = P(A)P(B)
CPSC 531: Probability Review
24
Independent Events (2)
 Example: Determine the probability pn that
exactly n tosses are required for a head to
appear for the first time.
Solution
We need to determine the probability of a
sequence of (n-1) tails followed by a head. The
probability of obtaining a heads or tails (for a
fair coin) is ½. Therefore, pn = (½)n.
CPSC 531: Probability Review
25
Law of Total Probability
 Let B1, B2, B3, …, Bk be mutually disjoint events such
that B1 U B2 U B3 U … Bk equal the sample space S.
Then, for any event A in S, we have
k
P( A)   P( B j ) P( A | B j )
Explanation
A = (B1A)U (B2A)U… (BkA).
The (BjA)’s are disjoint
events. Therefore, using
laws of conditional
probability we get:
k
k
j 1
j 1
P( A)   P( B j A)   P( B j ) P( A | B j ),
j 1
B1
B4
A
B2
B3
if P( B j )  0 for j  1,...,k
CPSC 531: Probability Review
26
Bayes’ Theorem
 Partition: The events B1, B2, B3, …, Bk form a
partition of a set S if they are mutually
disjoint and ik1 Bi  S
 Bayes’ Theorem: Suppose that B1, B2, B3, …, Bk
form a partition of sample space S such that
P(Bj)> 0 for j = 1, …, k. Let A be an event in S
such that P(A)>0. Then, for i = 1, …, k,
P( Bi ) P( A | Bi )
P( Bi | A)  k
 j 1 P( B j ) P( A | B j )
CPSC 531: Probability Review
27
Summary
 A review of basic probability and statistics
was presented. You probably enjoyed it!
 Next, we will talk about discrete and
continuous distributions.
CPSC 531: Probability Review
28