Download Introduction to Probability

Introduction to Probability
Classical Concept:
◦ requires finitely many and equally likely outcomes
◦ probability of event defined as number of favorable outcomes
(s) divided by number of total outcomes (N):
s
Probability of event =
N
◦ can be determined by counting outcomes
In many practical situations the different outcomes are not equally
likely:
◦ Success of treatment
◦ Chance to die of a heart attack
◦ Chance of snowfall tomorrow
It is not immediately clear how to measure chance in each of these
cases.
Three Concepts of Probability
◦ Frequency interpretation
◦ Subjective probabilities
◦ Mathematical probability concept
Elements of Probability, Jan 23, 2003
-1-
The Frequentist Approach
In the long run, we are all dead.
John Maynard Keynes (1883-1943)
The Frequency Interpretation of Probability
The probability of an event is the proportion of time that events
of the same kind (repeated independently and under the same
conditions) will occur in the long run.
Example:
Suppose we collect data on the weather in Chicago on Jan 21 and
we note that in the past 124 years it snowed in 34 years on Jan 21,
34
that is 124
100% = 27.4% of the time.
Thus we would estimate the probability of snowfall on Jan 21 in
Chicago as 0.274.
The frequency interpretation of probability is based on the following theorem:
The Law of Large Numbers
If a situation, trial, or experiment is repeated again and again, the
proportion of successes will converge to the probability of any one
outcame being a success.
Elements of Probability, Jan 23, 2003
-2-
0.2
0.4
0.6
0.8
Tosses 1 − 1000
0.0
Relative Frequency of Heads
1.0
The Frequentist Approach
0
100
200
300
400
500
600
700
800
900
1000
0.49
0.50
0.51
Tosses 1000 − 100000
0.48
Relative Frequency of Heads
0.52
Number of Tosses
1
10
20
30
40
50
60
70
80
90
100
0.500
Tosses 100000 − 1000000
0.495
Relative Frequency of Heads
0.505
Number of Tosses (in 1000s)
1
2
3
4
5
6
7
8
9
10
Number of Tosses (in 100000s)
Elements of Probability, Jan 23, 2003
-3-
The Subjectivist (Bayesian) Approach
Not all events are repeatable:
◦ Will it snow tomorrow?
◦ Will Mr Jones, 42, live to 65?
◦ Will the Dow Jones rise tomorrow?
◦ Does the Iraq have weapons of mass destruction?
To all these questions the answer is either “yes” or “no”, but we
are uncertain about the right answer.
Need to quantify our uncertainty about an event A:
Game with two players:
◦ 1st player determines p such that he will “win” $c · (1 − p) if
event A occurs and otherwise he will “loose” $c · p.
◦ 2nd player chooses c which can be positive or negative.
The Bayesian interpretation of probability is that probability
measures the personal (subjective) uncertainty of an event.
Example: Weather forecast
Meteorologist says that the probability of snowfall tomorrow is
90%.
He should be willing to bet $90 against $10 that it snows tomorrow
and $10 against $90 that it does not snow.
Elements of Probability, Jan 23, 2003
-4-
The Elements of Probability
A (statistical) experiment is a process of observation or measurement. For a mathematical treatment we need:
Sample Space S - set of possible outcomes
Example: An urn contains five balls, numbered from 1 through
5. We choose two at random and at the same time. What is the
sample space?
S = {1, 2}, {1, 3}, {1, 4}, {1, 5}, {2, 3}, {2, 4}, {2, 5},
{3, 4}, {3, 5}, {4, 5} .
Events A ⊆ S - an event is a subset of the sample space S
Example: In the example above the event A that two balls with
uneven numbers are choses is
A = {1, 3}, {1, 5}, {3, 5} .
Probability Function
P - assigns each A a value in [0, 1]
Example: Assuming that all events are equally likely we obtain
P(A) = 103 .
Elements of Probability, Jan 23, 2003
-5-
The Elements of Probability
Why not assign probabilities to outcomes?
Example: Spinner labeled from 0 to 1.
◦ Suppose that all outcomes s ∈ S = [0, 1) are equally likely.
◦ Assign probabilities uniformly on S.
P({s}) = c > 0 ⇒ P(S) = ∞
◦ P({s}) = 0 ⇒ P(S) = 0
◦
Solution: Assign to each subset of S a probability equal to the
“length” of that subset:
◦ Probability that the spinner lands in [0, 41 ) is 14 .
◦ Probability that the spinner lands in [ 12 , 34 ) is 14 .
◦ Probability that the spinner lands on 21 is 0.
In integral notation we have
Z b
P(spinner lands in [a, b]) = dx = b − a.
a
Remark:
Strictly speaking, we can define above probability only on a set A of subsets A ⊆ S which
however covers all important and for this class relevant subsets.
In the case of finite or countably infinite sample spaces S there are no such exceptions
and A covers all subsets of S.
Elements of Probability, Jan 23, 2003
-6-
A Set Theory Primer
A set is “a collection of definite, well distinguished objects of our perception
or of our thought”. (Georg Cantor, 1845-1918)
Some important sets:
◦
= {1, 2, 3, . . .}, the set of natural numbers
N
◦ Z = {. . . , −2, −1, 0, 1, 2, . . .}, the set of integers
◦ R = (−∞, ∞), the set of real numbers
Intervals are denoted as follows:
[0, 1] the interval from 0 to 1 including 0 and 1
[0, 1) the interval from 0 to 1 including 0 but not 1
(0, 1) the interval from 0 to 1 not including 0 and 1
If a is an element of the set A then we write a ∈ A.
If a is not an element of the set A then we write a ∈
/ A.
Suppose that A and B are subsets of S (denoted as A, B ⊆ S).
The empty set is denoted by ∅ (Note: ∅ ⊆ A for all subsets A of S).
Difference of A and B (A\B): Set of all elements in A which are not in B.
Intersection of A and B (A ∩ B): Set of all elements in S which are both
in A and in B.
Union of A and B (A ∪ B): Set of all elements in S that are in A or in B.
Complement of A (A{ or A0 ): Set of all elements in S that are not in A.
Note that A ∩ A{ = ∅ and A ∪ A{ = S
A and B are disjoint if A and B have no common elements, that is A∩B =
∅. Two events A and B with this property are said to be mutually
exclusive.
Elements of Probability, Jan 23, 2003
-7-
The Postulates of Probability
A probability on a sample space S (and a set A of events) is a
function which assigns each subset A a value in [0, 1] and satisfies
the following rules:
Axiom 1: All probabilities are nonnegative:
P(A) ≥ 0
for all events A.
Axiom 2: The probability of the whole sample space is 1:
P(S) = 1.
Axiom 3 (Addition Rule): If two events A and B are mutually exclusive then
P(A ∪ B) = P(A) + P(B),
that is the probability that one or the other occurs is the sum
of their probabilities.
More generally, if countably many events Ai, i ∈ N are mutually exclusive (i.e. Ai ∩ Aj = ∅ whenever i 6= j) then
P
∞
S
i=1
Ai =
∞
P
i=1
Elements of Probability, Jan 23, 2003
P(Ai).
-8-
The Postulates of Probability
Classical Concept of Probability
The probability of an event A is defined as
P(A) = #A
,
#S
where #A denotes the number of elements (outcomes) in A.
It satisfies
◦ P(A) ≥ 0
◦
P(S) = #S/#S = 1
◦ If A and B mutually exclusive then
P(A ∪ B) = #(A#S∪ B)
=
#A #B
+
= P(A) + P(B).
#S #S
Elements of Probability, Jan 23, 2003
-9-
The Postulates of Probability
Frequency Interpretation of Probability
The probability of an event A is defined as
n(A)
P(A) = n→∞
lim
,
n
where n(A) is the number of times event A occurred in n repetitions.
It satisfies
◦ P(A) ≥ 0
◦
P(S) = limn→∞ nn = 1
◦ If A and B mutually exclusive then n(A ∪ B) = n(A) + n(B).
Hence
n(A ∪ B)
P(A ∪ B) = n→∞
lim
n
n(A)
n(B) = lim
+
n→∞
n
n
n(A)
n(B)
= lim
+ lim
= P(A) + P(B).
n→∞ n
n→∞ n
Elements of Probability, Jan 23, 2003
- 10 -
The Postulates of Probability
Example: Toss of one die
The events A = {1} and B = {4 5} are mutually exclusive.
Since all outcomes are equiprobable we obtain
P(A) = 16
and
P(B) = 13 .
The addition rule yields
P(A ∪ B) = 16 + 13 = 36 = 12 .
On the other hand we get for C = A ∪ B = {1 4 5}
P(C) = 36 = 12 .
The first two axioms can be summarized by the
Cardinal Rule: For any subset A of S
0 ≤ P(A) ≤ 1.
In particular
◦ P(∅) = 0
◦
P(S) = 1
Elements of Probability, Jan 23, 2003
- 11 -