Download Introduction to Probability to accompany Introduction Statistical

Introduction to Probability
to accompany
Introduction
Statistical decision making – estimation (Chapter 6 of the text) and hypothesis testing (Chapters 7
to 10) – is based on probability theory, and this supplement provides an introduction to this field.
People have been interested in probability and have played games of chance for thousands of years
but it was not until the 17th century that Blaise Pascal, Pierre deFermat, Christian Huygens, and
others outlined the beginnings of systematic study of probabilities – the mathematical science of
estimating the likelihood of a particular event or outcome such as roll of the dice or draw of a card.
In this supplement we shall examine some basic laws of probability. We will look at the methods of
calculating probabilities and combinations and permutations. This supplement provides a more
detailed and methodical treatment of probability than it was possible to include in the text. It should
be read in conjunction with Chapter 4, specifically Section 4.6 on “Using the Normal Curve to
Estimate Probabilities.” This material was adapted from Elementary Statistics for the Social
Sciences by Professor J. Anthony Capon.
Introduction to Probability
Probability may be defined as the frequency with which a particular event will occur in an infinite
number of trials. Probability is usually expressed as a proportion. If we let Y be the number of
trials, and X be the number of times a specified event E occurs, then the probability of event E
occurring may he expressed as
Formula 1 P[ E ] 
X
Y
An example will clarify how to calculate a probability. Suppose we have a new penny. If we flip
the coin, we will get a head (one event) or tail (another event). If we flipped the coin a very large
number of times, we would get H number of heads and T number of tails, assuming the coin cannot
land on its edge. We could express the probability of getting a head – P[H] – as
Formula 2 P[ H ] 
H
H T
If the coin always came up heads, the probability of getting a head would be 1.00. If it never came
up heads, the probability would be 0. Therefore, the maximum value a probability can be is 1; the
minimum is 0. But you do not have to be a statistician to know that if you flipped the coin a very
large number of times, you would get heads about half the time. Therefore, the probability of
getting a head would be about .5. Note that we require the coin to be flipped a very large number
of times, approaching infinity. We are not saying that if you flipped the coin 10 times, you would
get exactly 5 heads. What we are saying is that as the number of trials increases, the proportion of
heads will approach .5. Theoretically, if the number of trials is infinite, the proportion of heads will
be .5.
1
Copyright © 2013 by Nelson Education Ltd.
The type of probability we have been discussing so far is called a priori probability. It is based on
mathematical reasoning and logic. We cannot demonstrate that the probability of getting a head
when flipping a coin is .5, since by definition we could not flip the coin an infinite number of
times. But we can show that probability is .5 logically.
Another type of probability is called an empirical probability, which is determined by observation
and experimentation. Death rates, the probabilities of death in a given year for a given age and sex,
are based on empirically obtained mortality data. Insurance company statisticians use these
probabilities to calculate life insurance rates. While both types of probability are useful, the type of
decision making with which we shall be concerned in this text is based on a priori probability.
Calculating Probabilities
There are two basic rules used in the calculation of probabilities: the addition rule and the
multiplication rule. The addition rule deals with the probability of one of a number of possible
events occurring, usually in one trial. The multiplication rule deals with the probability of more
than one event occurring, usually in more than one trial.
The Addition Rule
When we apply the addition rule for probability, we want to know the probability of event A or
event B occurring. We may symbolize this probability as P[A or B]. In order to calculate P[A or B]
we first determine whether events A and B are mutually exclusive. Two events are mutually
exclusive if they cannot occur in the same trial. For example, if we flip a coin, we get either a head
(event A) or a tail (event B); we cannot get both in the same trial. Therefore, events A and B are
mutually exclusive in this case. If we draw one card from a deck of playing cards, we cannot get
both a spade and a heart. So these two events are also mutually exclusive. On the other hand, if we
drew one card, we could get both a spade and a 10 (the 10 of spades). These two events are not
mutually exclusive, since both could occur in one trial.
If two events are mutually exclusive, the probability of one or the other of them occurring in the
same trial is the sum of their respective probabilities. The addition rule – for mutually exclusive
events – can be stated as:
Formula 3
P[A or B] = P[A] + P[B]
We will illustrate the use of this formula with several examples. First, consider the probability of
getting a head or a tail in one flip of a coin. Using the addition rule:
P[H OR T] = P[H] + P[T]
= 0 5 + 0.5
= 1.00
You will always get a head or a tail when you flip a coin.
2
Copyright © 2013 by Nelson Education Ltd.
Second, suppose we have an urn in which there are eight red marbles, five blue marbles, and three
yellow marbles. If we draw one marble, what is the probability that it will be blue or yellow?
There are 16 marbles in the urn. The probability of drawing a blue marble (P[B]) is 5/16. The
probability of drawing a yellow marble (P[Y]) is 3/16. Therefore, since these events are mutually
exclusive, we can apply the addition rule:
P[B OR Y] = P[B] + P[Y]
= 5/16 + 3/16
= 8/16
= 0.5
As a third example, consider the probability for drawing a red king or a spade in a single draw from
a deck of cards. First, the two events are mutually exclusive, since there are no red kings that are
also spades. There are 52 cards in a deck, including 2 red kings and 13 spades. Therefore,
P[red K or S] = P[red K] + P[S]
= 2/52 + 13/52
= 15/52
= 0.29
Extending the Addition Rule to Two Events that are not mutually exclusive. Now suppose that
two events are not mutually exclusive. Then the probability of getting event X or event Y is the
sum of their respective probabilities minus the probability of both events occurring at the same
time. In notation,
Formula 4
P[A OR B] = P[A] + P[B] – P[A & B]
Where P[A&B] is the probability of events A and B occurring at the same time. We can illustrate
with an example. If we draw one card from a deck of cards, what is the probability that it will be a
club or a face card? We can calculate the probability by counting the cards. First, we have 13 clubs
(including the jack, queen, and king of clubs). We also have 9 other face cards (excluding the jack,
queen, and king of clubs). Therefore, since there are 22 cards in the deck that are either clubs [C]
or face cards [FC],
P[C OR FC] = 22/52 = 0.42
Now, let’s repeat the calculation using the formula. There are 13 clubs in the deck, 12 face cards
(jack, queen, and king of each of the four suits), and 3 clubs that are also face cards (jack, queen
and, king of clubs) Therefore,
P[C OR FC] = P[C] + P[FC] – P[C&FC]
= 13/52 + 12/52 – 3/52
= 22/52
= 0.42
3
Copyright © 2013 by Nelson Education Ltd.
Extending the Addition Rule to Three Mutually Exclusive Events. The addition rule may be
extended to three or more events. If we have three mutually exclusive events, probability of any
one of them occurring is simply the sum of their respective probabilities:
Formula 5
P[A OR B OR C] = P[A] + P[B] + P[C]
For example, if we draw one card from a deck of cards, what is the probability that it will be a 3, a
red 10, or a black face card? The three events are mutually exclusive. In the deck there are four 3s,
two red 10s, and six black face cards. Therefore,
P[3 or red 10 or black FC] = P[3] + P[red 10] + P[black FC]
= 4/52 + 2/52 + 6/52
= 12/52
= 0.23
We can extend the addition rule to any number of mutually exclusive events. In general, for any
number of mutually exclusive events, the probability of any one of them occurring is the sum of the
probabilities of each event.
Extending the Addition Rule to Three Events that are not Mutually Exclusive. Let us now
consider the probability of more than two events that are not mutually exclusive. Extending the
addition rule to cover this circumstance is not nearly as straightforward as the case in which the
events are mutually exclusive. Since the events overlap with each other in three combinations (A
with B, A with C, B with C), we have to subtract the overlap of events three times. However, we
also have to take account of the overlap of all three events (A with B with C) and add it back into
the final calculation. So, to find the probability of three events which are not mutually exclusive,
we must use Formula 6:
Formula 6
P[A OR B OR C] = P[A] + P[B] + P[C] - P[A&B] - P[A&C] –
P[B&C] + P[A&B&C]
Let us consider an example. Suppose we draw one card from a deck. What is the probability that it
will be a red face card [FC], a king [K], or a diamond [D]? Note that these events are not mutually
exclusive since there is a red face card that is a king, red face cards that are also diamonds, and a
king that is also a diamond.
First, let us figure out the answer simply by counting cards. There are six red face cards – the
jacks, queens, and kings of hearts and diamonds. There are two kings we have not already counted
– the kings of spades and clubs. There are 10 diamonds we have not already counted – the ace
through 10 of diamonds. Therefore, there are 18 cards that fit our criteria, so the probability of
drawing any one of them must be 18/52 or 0.35.
4
Copyright © 2013 by Nelson Education Ltd.
Let us now use the formula:
P[red FC or K or D] = P[red FC] + P[K] + P[D] – P[red FC & K]
– P[red FC & D] – P[K & D] + P[red FC & K & D]
Where:
P[red FC] =
P[K] =
4
52
P[D] =
13
52
6
52
P[red FC & K] =
2
(kings of hearts and diamonds)
52
P[red FC & D] =
3
(jack, queen, and king of diamonds)
52
P[K & D] =
1
(king of diamonds)
52
P[red FC & K & D] =
1
(king of diamonds)
52
Therefore,
P [red FC or K or D] = 6/52 + 4/52 + 13/52 - 2/52 – 3/52 – 1/52 + 1/52
= 18/52
= 0.35
The addition rule may also be extended to four or more events that are not mutually exclusive.
However, the calculations become progressively more difficult, and we will not do so here.
The Multiplication Rule
As we have seen, the addition rules is used when we want to determine the probability of event A
or event B occurring in one trial. The multiplication rule is used when we want to determine the
probability of event A followed by event B, in two trials. Just as with the addition rule, we have
two cases; one when the events are independent, and the other when the events are conditional.
Two events are independent if the occurrence of one event in no way affects the probability of the
second event occurring. For example, if we flip a coin twice, what is the probability of getting a
head followed by a tail? The probability of getting a head on the first flip is 1 out of 2 or 0.50.
5
Copyright © 2013 by Nelson Education Ltd.
Regardless of what we get on the first flip, the probability of getting a tail on the second flip is also
1 out of 2 or 0.50. So the two events are independent. Similarly, suppose we roll a die twice.
What is the probability of getting a 2 followed by a 4? What we get on the first roll in no way
affects what we get on the second roll. Both probabilities are 1 out of 6 or 0.17, and the events are
independent.
Now suppose we have a bowl of fruit containing 4 apples, 3 peaches, and 2 pears. If you are going
to eat two randomly selected pieces of fruit from this bowl, what is the probability that they will be
a peach followed by an apple? Since there are 9 pieces of fruit in the bowl, the probability that the
first fruit selected will be a peach is 3 out of 9 or 0.33. Once the peach is eaten, however, there are
only 8 pieces of fruit left, of which 4 are apples. The probability that the second fruit will be an
apple then is 4 out of 8, not 4 out of 9. So the probability of selecting an apple as the second fruit is
conditional upon the selection of the peach, since the first event (the selection of the peach) affects
the probability of the second event (the selection of the apple).
If two events are independent, the probability of the first event occurring followed by the second
event is the product of their respective probabilities. In notation, if we let [A&B] represent A
followed by event B, then:
Formula 7
P [A&B] =P [A] * P [B]
The probability of getting a head on the first flip of a coin followed by a tail on the second flip is:
P [H&T] =P [H] * P [T] = (0.50) * (0.50) = 0.25
The probability of getting a 2 followed by a 4 in two rolls of a die is:
P [2&4] = P [2] * P [4] = (0.17) (0.17) = 0.03
Now suppose we draw two cards from a deck of cards. What is the probability of getting a face
card followed by a 10? Are the two events independent or conditional? The answer depends on
what we do with the first card after we draw it. If we put it back in the deck, there will still be 52
cards in the deck, and the two events will be independent. If we do not put it back, however, there
will be only 51 cards remaining for the second draw, and the events will be conditional.
When we put the first card back, we call it drawing (sampling) with replacement. If we do not
put the first card back, we call it drawing (sampling) without replacement. Suppose we draw
with replacement, and thus have independent events. Since there are 12 face cards and 4 tens:
P [FC&10] = P [FC] * P [10]
= (12/52) * (4/52)
= (0.2308) * (0.0769)
= 0.0178
6
Copyright © 2013 by Nelson Education Ltd.
If two events are conditional, then the probability of the first event occurring followed by the
second event is obtained by multiplying the probability of the first event by the probability of the
second event given that the first event has occurred. If P [B/A] means “the probability of B
occurring given that A has occurred,” then:
P [A&B] = P [A] * P [B/A]
If we draw the two cards without replacement, the probability of the first card being a face card
remains 12/52. But since there will be only 51 cards remaining in the deck for the second draw, the
probability of the second card being a 10 is 4/51. Therefore,
P [FC&10] = P [FC] P * [10/FC]
= (12/52) * (4/51)
= (0.2308) * (0.0784)
= 0.0181
Let us return to the bowl of fruit, with 4 apples, 3 peaches, and 2 pears. Since we are drawing
without replacement, the probability of getting a peach followed by an apple is:
P [peach & apple] = P[peach] * P [apple/peach]
= (3/9) * (4/8)
= (0.33) * (0.50)
= 0.17
One final note on conditional probabilities:
P [A] P [B/A] = P [B] P [A/B]
In the previous example, this tells us that the probability of getting a peach followed by an apple is
the same as the probability of getting an apple followed by a peach:
P[peach] P [apple/peach] = P[apple] P [peach/apple]
(3/9)(4/8)= (4/9) (3/8)
12/72=12/72
Extending the Multiplication Rule to Three or More Events. Extending the multiplication rule
to three or more events is straightforward. If three or more events are independent, then the
probability of the events occurring in a particular order is the product of their representative
probabilities:
Formula 8 P [A&B&C] = P [A] * P [B] * P[C]
7
Copyright © 2013 by Nelson Education Ltd.
If an urn contains five red, four green, and two yellow marbles, then the probability of selecting a
red marble, followed by a yellow and a green in that order, with replacement, is:
P[R&Y&G] = P[R] P[Y] P [G]
= (5/11) * (2/11) * (4/11)
= (0.45) * (0.18) * (0.36)
= 0.03
If the events are conditional, then the probability of three events occurring in a particular order is:
Formula 9
P [A&B&C] = P [A] * P [B/A] * P[C/ (B&A)]
Where P[C/ (B&A)] is the probability of event C occurring given that events B and A have
occurred.
Suppose we draw the marbles without replacement in the first example of the section. Then:
P[R&Y&G] = P[R] * P[Y/R] * P [G/(Y&R)]
After we draw the red marble, there will be 10 marbles remaining. After we then draw the yellow
marble, there will be 9 remaining. So:
P[R] P[Y/R] P [G/(Y&R)] = (5/11) * (2/10) * (4/9)
= (0.46) * (0.20) * (0.44)
= 0.04
The multiplication rules can be extended directly to any number of independent or conditional
events.
Combinations and Permutations
To this point we have been making an assumption that simplifies our calculations considerably.
With the multiplication rule, we have assumed that we want to calculate the probability of getting
event A followed by event B. In other words, we have assumed that the order in which the events
occurred was important. But in many circumstances we may be less interested in the order in
which events occurred and more interested in whether they occurred at all. For example, a bridge
player might be interested in the probability that his right-hand opponent was dealt both the king
and the queen of trumps. It is quite irrelevant whether the king was dealt before the queen or the
queen before the king.
When we are interested in the occurrence of a set of events regardless of order, we refer to this as a
combination. When we are interested in the order of a set of events, we refer to this as a
permutation.
8
Copyright © 2013 by Nelson Education Ltd.
Consider an example. Suppose we have three cards: a king (K), a queen (Q), and a jack (J). If we
are going to pick two cards at a time, and order is important, how many permutations can we get?
There are six:
KQ, QK, KJ, JK, QJ, JQ.
But if order is not important, then KQ is the same as QK, KJ the same as JK, and QJ the same as
JQ. So there are only three combinations:
KQ (or QK),
KJ (or JK),
QJ (or JQ)
It is often useful to know how many combinations or permutations of events can occur.
Fortunately, there are relatively simple formulae that give us the answers. If we have n things, and
we wish to take r of them at a time, then the number of combinations of n is given by:
Cr 
Formula 10
n
n!
r !(n  r )!
The symbol n! (“n factorial”) means: n(n-1) (n-2)(n-3)…[n-(n-1)]
Thus, for example,
7!= 7 * 6 * 5 * 4 * 3 * 2 * 1 = 5,040
So if we have three cards (n), how many combinations (r) of two of them can we make?
n!
r !(n  r )!
3!
nCr 
2!(3  2)!
3  2 1
nCr 
2  1 1
6
nCr 
3
2
Cr 
n
As we saw with the three cards, there were indeed three combinations when we took the cards two
at a time.
9
Copyright © 2013 by Nelson Education Ltd.
Let’s consider another example. If we have seven persons, and are going to pick four of them for a
committee, how many choices do we have? Since the order in which we pick the members of the
committee is not important, we are asking how many combinations (r) of four persons we can
make from a group of seven (n):
n!
r !(n  r )!
7!
nCr 
4!(7  4)!
7  6  5  4  3  2 1
nCr 
4  3  2  1 3  2 1
5040
nCr 
 35
144
Cr 
n
There are 35 different committees of four persons that can he made from a group of seven.
When the order of things is important, we have a permutation, and the formula for the number of
permutations of n things taken r at a time is:
Formula A11
Pr 
n
n!
(n  r )!
For the three cards in the previous example, the number of permutations of three cards (n) taken
two (r) at a time is:
n!
(n  r )!
3!
3P2 
(3  2)!
3  2 1
3P2 
6
1
n
Pr 
And as we have already seen, there were six permutations of three cards taken two at a time.
10
Copyright © 2013 by Nelson Education Ltd.
If the order were important in the example where we were selecting four persons from seven for a
committee, we would have:
n!
(n  r )!
7!
3P2 
(7  4)!
7  6  5  4  3  2 1
3P2 
 840
3  2 1
n
11
Pr 
Copyright © 2013 by Nelson Education Ltd.