Download 1 Probability Theory

1
Probability Theory
1.1
Experiment, Outcomes, Sample Space
Example 1 An psychologist examined the response of people standing in line at a copying
machines.
Student volunteers approached the person first in line and asked for permission to skip line,
”because I only have 5 pages to copy, and I am really in a rush”.
The person at the front of the line, could now give permission or not.
This is an example for an experiment! Where permission yes/no are the possible outcomes.
The set of all possible out comes is called the sample space, S = {yes, no}.
Definition:
1. The sample space S is the set of all possible outcomes.
2. An event is a set of one or more outcomes for an experiment. It is a sub set of S.
3. An event that includes one and only one of the outcomes is called a simple event.
4. A compound event is an event that includes at least 2 outcomes for an experiment.
Examples:
Experiment
Outcomes
Sample Space
toss a coin
Head, Tail
S = {H, T}
roll a die
1,2,3,4,5,6
S = {1, 2, 3, 4, 5, 6}
toss 2 coins HH, HT, TH, TT S = {HH, HT, TH, TT}
Use Venn Diagrams to illustrate experiments.
The outer box represents the sample space, which contains all possible outcomes: E1 , . . . , E6 .
Is A = {E1 , E2 } the collection of outcomes the appropriate outcomes are circled and labelled
with the letter A.
S
E3
A
E1
E4
E5
E2
E6
Or use Tree Diagrams to illustrate experiments.
We start at one point we draw a branch for each possible outcome.
1
Tree diagram are especially useful for multiple step experiments
Example 2
Tossing two coins, then S = {HH, HT, T H, T T }
Example 3 Consider the experiment of tossing a coin, then S = {1, 2, 3, 4, 5, 6}. Consider
the simple event of rolling a 6. E = {6} and the compound event of rolling an even number
A = {2, 4, 6}.
1.2
Calculating Probabilities
Definition 1 The Probability of an event is a numerical measure of the likelihood that the
event will occur.
Probability Rules:
2
1. The probability is always a number between 0 and 1. – For all events A, 0 ≤ P (A) ≤ 1.
2. The sum of the probabilities of all simple events for an experiment
is always 1.
∑
– Let Ei denote the simple events of an experiment, then
P (Ei ) = 1
The probability for the impossible event is zero and the probability of the sure event is one.
Example 4 Assume an experiment with equally likely outcomes (like rolling a fair die).
Then the probability of each simple event should be the same and those probabilities need to
add up to one, so for all simple events Ei
P (Ei ) =
1
total number of possible outcomes
If you roll a fair die all outcomes are equally likely, there is a total of 6 possible outcomes, so
P ({1}) = . . . = P ({6}) = 1/6
In this case the probability of a compound event is calculated by
P (A) =
number of outcomes favourable to A
total number of possible outcomes
The probability event, A, to roll an even number with a fair die is P (A) = 3/3 = 1/2
But how do we assign probabilities in the cases where the outcomes do not have the same
probabilities?
Relative frequency interpretation of probability
The probability of an event E, denoted as P (E), is the long-run proportion of the time that the
event will occur when the experiment is performed repeatedly under identical conditions. This
is, if n is the number of same experiments and the event E occurred fn times in n experiments,
then
P (E) = n→∞
lim
fn
n
This property is called the law of large numbers.
Example 5 Assume there 6 of the 60 people enrolled in this class participate in competitive
sports.
Let A be the event that a randomly chosen person from this class participates in sports, then
P (A) = 6/60 = 0.1 (using the the rule equally likely outcomes).
But we also could conduct the experiment of picking randomly a person from the class, determine sport yes/no, and by keeping doing this we might find 5 ”yes” after 40 people being
asked and ”estimate” 5/40=1/8=0.125 as the probability.
Or do it 1000 times and get 102 ”yes” and estimating 102/1000=0.102.
3
1.2.1
Probability Rules
Definition:
• The union of events A and B, denoted by A or B, is the event that either A or B or
both occur.
A
B
• The intersection of events A and B, denoted by A and B, is the event that both A and
B occur.
A
B
• The complement of an event A, denoted by Ā = Ac , is the event that A does not occur.
Ac
A
Example: Continue the die rolling example.
C is the event to roll an odd number. Let D be the event to roll a number smaller than
4.
Then D = {1, 2, 3}.
The probability of every simple event is 1/6.
With the definition follows that P (C) = P ({2}) + P ({4}) + P ({6}) = 1/6 + 1/6 + 1/6 =
3/6 = 1/2.
4
P (C or D) = P ({1, 3, 5, 2}) = 4/6 = 2/3.
P (C and D) = P ({1, 3}) = 2/6 = 1/3.
Addition Rule
Let A and B be two events. The probability of the union can then be calculated by:
P (A or B) = P(A ∪ B) = P(A) + P(B) − P(A and B)
Remark:
The subtraction of P (A and B) is necessary because this area is counted twice by the addition
of P (A) and P (B), once in P (A) and once in P (B). Check the diagram below.
B
A
A and B
Rule for Complements
The probability of the complement of an event is 1 minus the probability of this event.
Mathematically:
If P (Ac ) = 1 − P (A)
Example:
Let A = {1, 2} and B = {1, 3, 5}, with P (A) =
P ({1}) = 61 .
1
, P (B)
3
=
1
,
2
and P (A and B) =
With the Addition Rule we get P (A or B) = P (A) + P (B) − P (A and B) = 1/3 + 1/2 −
1/6 = 2/3.
With the Rule for Complements is P (Ac ) = 1 − 1/3 = 2/3.
1.3
Independence of Two Events and Conditional Probabilities
Two events are considered independent, when the occurrence of one of the events has no impact
on the probability for the second event to occur. They don’t have an impact on each other.
In this section we will define and do examples to illustrate this concept. For defining the
independence of events properly we first introduce the concept of a conditional probability.
5
Definition
If A and B are events with P (B) > 0, the conditional probability of A given B is defined by
P (A|B) :=
P (A and B)
.
P (B)
Bc
B
S
A and B
c
A and B
A
The interpretation of the conditional probability P (A|B) is as follows:
Given that you know already event B occurred, what is the probability that A occurs.
Example:
Consider the event B to roll an even number with a fair 6 sided die.
The Probability for the event A to roll a 1, given B equals 0: P (A|B) = 0
Because: If you roll a die, someone peeks and tells you that the number is even, than this
number can not be a 1. So the conditional probability of A given B must be 0.
In probability theory two event are considered independent, if the knowledge that one of them
has occurred does not change the probability for the second to occur.
Definition:
Events A and B, with P (B) > 0, are independent if
P (A|B) = P (A),
(which is equivalent to P (A and B) = P (A) · P (B)).
Remark:
From this definition we also learn that: If we know that two events A and B are independent
then
P (A and B) = P (A)P (B)
This is only true for INDEPENDENT events.
The following example shows how to apply these two two concepts to the experiment of rolling
a die.
Example:
Consider the experiment to roll a fair die.
Let A = {1, 2, 3}
B = {4}
C = {3, 4}
Then we can calculate, by finding the simple events contained by the described events and
adding their probabilities:
6
• P (A) =
1
2
• P (B) =
1
6
• P (C) =
1
3
• P (A ∩ B) = 0
• P (A ∩ C) =
• P (B|A) =
1
6
P (B and A)
0
= 1 = 0 (by definition of the conditional probability)
P (A)
2
Since P (B) ̸= P (B|A), we find that B and A are not independent (apply the definition
for independence).
• P (C|A) =
P (C and |A)
=
P (A)
1
6
1
2
=
1
3
Since P (C) = P (C|A), we find that C and A are independent.
Example 6
In a company all 100 employees were asked whether they are in favour of or against paying
high salaries to CEOs of large companies.
The following table show the results depending on the gender of the employees
In Favour Against
Male
15
45
Female
4
36
In order to complete this table we should add row and column totals
In Favour Against
Male
15
45
Female
4
36
Total
19
81
Total
60
40
100
Suppose one employee is selected at random, we can ask what is the probability that the
employee is a male (female)? 60/100 (40/100).
We say that theses numbers give the marginal probabilities of the variable gender. We
only consider one variable, and neglect the other .
The marginal probabilities of the variable ”response” is: P(In Favour)=19/100, P(Against)=81/100
Suppose now we select randomly an employee and ask, given the person is a male, what is the
probability is that he is in favour?
This is asking for the conditional probability of ”In Favour” given ”male”:
P(In Favour | male)= 15/60=0.25
and
P(In Favour | female)= 4/40=0.1
7
Example:
A satellite has two power systems, a main and an independent backup system.
Suppose the probability of failure in the first ten years for the main system is 0.05 and for the
backup system 0.08.
What is the probability that both systems fail in the first 10 years and the satellite will be
lost?
Let M be the event that the main system fails and B the event that the backup systems fails.
Since the systems are independent we get:
P (M ∩ B) = P (M ) · P (B) = 0.05 · 0.08 = 0.004
The event O that at least one of the systems is still operational after 10 years is the complement
event of both systems failing. We calculate:
P (O) = 1 − P (Oc ) = 1 − 0.004 = 0.996
The following example shows the practical relevance of these concepts. In this example they
are applied to an AIDS test, but please be aware that the same arguments apply to other
clinical tests.
Example:
• ELISA is a test for detecting HIV antibodies in blood samples.
• The test is positive in 99.7% of blood samples containing antibodies.
• The test is positive in 1.5% of blood samples containing no antibodies.
Assuming that 0.18% of the population do have HIV antibodies in their blood (Statistics
Canada for 2005), how big is the probability that a person has HIV if his/her blood tests
positive?
For finding an answer to this question we translate the number given above into probability
theory.
First choose the denotation:
• Let H+ be the event of a blood sample containing antibodies.
• Then is H− = H+C the event of a blood sample not containing antibodies.
• Let E+ be the event of a positive ELISA test result.
• Then is E− = E+C the event of a negative ELISA test result.
Now “translate” the facts stated above into conditional probabilities.
We get from the introductory statements and the definition:
P (E+ |H+ ) = 0.997, P (E+ |H− ) = 0.015, and P (H+ ) = 0.01 and want to know P (H+ |E+ ).
We solve the problem by giving the table showing the probabilities for events H+ , H− and
E+ , E−
8
H+
H−
E+
E−
Total P (H+ ) = 0.0018
Total
P (H− ) = 1 − P (H+ ) = 0.9982 1
The entries in this table are P (E+ and H+ ), P (E+ and H− ), P (E− and H+ ), and P (E− and
H− )
Using the definition:
P (E+ |H+ ) =
P (E+ and H+ )
P (H+ )
so we get P (E+ and H+ ) = P (E + |H+)P (H+ ) and similarly
P (E+ and H− ) = P (E + |H−)P (H− )
After this we just find row totals and use the knowledge of column totals to deduct the values
still missing.
H+
E+
P (E+ andH+ )
=(0.997)0.0018
=0.0017946
E−
0.0000054
Total 0.0018
H−
P (E+ andH− )
=(0.015)0.9982
=0.014973
0.983227
0.9982
Total
0.0167676
0.9832324
1
From the table we now get:
P (H+ |E+ ) =
P (H+ and E+ )
0.0017946
=
≈ 0.1070
P (E+ )
0.0167676
This probability is very low and a test result would have to be confirmed with additional tests.
This shows that ELISA is rather a test to exclude HIV then to test for HIV. Try on your own
to calculate P (H− |E− ).
9