Download Statistical Foundations: Elementary Probability Theory

Statistical Foundations: Elementary
Probability Theory
Lecture 2
August 29, 2006
Psychology 790
Lecture #2 - 8/29/2006
Slide 1 of 24
Today’s Lecture
Overview
➤ Today’s Lecture
➤ Where this Fits
Probability
Wrapping Up
Lecture #2 - 8/29/2006
●
Probability (Hays, Chapter 1).
●
Events.
●
Sets (Hays, Appendix E).
●
Gambles.
●
Simple statistical inference.
●
Fun.
Slide 2 of 24
The Big Picture
●
Overview
➤ Today’s Lecture
➤ Where this Fits
Last time we talked about levels of measurement, which was
introduced to make you understand that statistics cannot be
applied to any and all types of numbers.
✦
Probability
Only numbers with certain properties can be used with
certain statistical methods.
Wrapping Up
Lecture #2 - 8/29/2006
●
Today, we introduce probability.
●
As you can imagine, we will end up using concepts in
probability quite often, particularly in tests of statistical
hypotheses.
●
Although it may be hard to make the connection, the topics
of today’s class will lay the foundation for the statistics we will
conduct throughout this semester (and the next).
Slide 3 of 24
Introduction to Probability
Overview
Probability
➤ Introduction
➤ Experiments
➤ Events
➤ Probabilities
➤ 5 Simple Rules
➤ Equally Probable
Events
➤ Long Run
➤ Odds
➤ Conditional
Probability
➤ Independence
➤ Dependence
➤ Bayes’ Theorem
Wrapping Up
Lecture #2 - 8/29/2006
●
Statistical inference involves statements about probability.
●
Much work in probability theory evolved from trying to model
games of chance - such as roulette.
●
Science has adopted probability to describe the likelihood of
events under study.
✦
Scientists are the most addicted gamblers out there.
●
If you are one to like gambling, then consider what goes on
in scientific studies to be a lot like playing your favorite casino
game.
●
Science tries to lay out the odds that a given behavior is true
by observing the behavior and making deductions about
what should be true in the long run.
Slide 4 of 24
Simple Experiments
●
Overview
Probability
➤ Introduction
➤ Experiments
➤ Events
➤ Probabilities
➤ 5 Simple Rules
➤ Equally Probable
Events
➤ Long Run
➤ Odds
➤ Conditional
Probability
➤ Independence
➤ Dependence
➤ Bayes’ Theorem
✦
●
Wrapping Up
●
Lecture #2 - 8/29/2006
To go in-depth with the basics of probability, we need to
define the concept of a simple experiment.
A simple experiment is some well-defined act or process
that leads to a well-defined outcome.
Examples of simple experiments:
✦
Tossing a coin - leads to heads or tails.
✦
Rolling a die - seeing what number comes up.
✦
Asking a person about their political preference.
✦
Giving a person an intelligence test and calculating their
score.
The outcomes of experiments, as you could guess, are the
things we often assign numbers to - remember last lecture...
Slide 5 of 24
Events
●
The basic elements of probability theory are the possible
distinct outcomes of a simple experiment.
Probability
➤ Introduction
➤ Experiments
➤ Events
➤ Probabilities
➤ 5 Simple Rules
➤ Equally Probable
Events
➤ Long Run
➤ Odds
➤ Conditional
Probability
➤ Independence
➤ Dependence
➤ Bayes’ Theorem
●
The set of all possible outcomes is called the sample space.
Wrapping Up
●
Overview
Lecture #2 - 8/29/2006
✦
●
All statistical distributions have a well-defined sample
space (which is just as important to recognize as the
appropriate level of measurement).
Any member of the sample space is called a sample point or
an elementary event.
✦
Having a coin land on “heads” is an example of a sample
point.
Events are sets, or classes, having the elementary events as
members.
Slide 6 of 24
Events as Sets of Probabilities
●
Just to conform to the book, consider the symbol S as the
sample space - a set that contains every possible event.
●
We will define ∅ as the null event - an event that cannot
happen
Overview
Probability
➤ Introduction
➤ Experiments
➤ Events
➤ Probabilities
➤ 5 Simple Rules
➤ Equally Probable
Events
➤ Long Run
➤ Odds
➤ Conditional
Probability
➤ Independence
➤ Dependence
➤ Bayes’ Theorem
Wrapping Up
Lecture #2 - 8/29/2006
✦
We also refer to ∅ as the empty set, which is a part of
every set S.
●
Suppose we have two events, A and B, each composed of
elementary events in S, and thus each are subsets of S.
●
We can talk about having an event which is part of sets A
and B (or part of set A or set B).
●
I can tell this will quickly spiral out of control into mass
boredom, so let me just say that Venn diagrams are useful
tools to help describe what we are talking about with sets.
Slide 7 of 24
Sets with Diagrams
●
This slide has a few things we will distinguish with sets - I will
draw along with these.
●
We have the sample space S.
●
We have event A.
●
Everything that is not a part of event A is denoted as ∽ A
●
We have event B.
●
All events that are part of A and B are said to fall into the
intersection of A and B, or (A ∩ B).
●
All events that are part of A or B are said to fall into the
union of A and B, or (A ∪ B).
●
Events that cannot happen are part of ∅.
Overview
Probability
➤ Introduction
➤ Experiments
➤ Events
➤ Probabilities
➤ 5 Simple Rules
➤ Equally Probable
Events
➤ Long Run
➤ Odds
➤ Conditional
Probability
➤ Independence
➤ Dependence
➤ Bayes’ Theorem
Wrapping Up
Lecture #2 - 8/29/2006
Slide 8 of 24
Probabilities
●
Probabilities are the numbers assigned to each and every
event in some sample space.
●
We denote the probability of an event A as p(A).
●
For each type of set relation we just covered, we can attach
a probability: for instance p(A ∪ B).
●
Modern probability theory is constructed from a small
number of basic statements about the properties these
numbers must exhibit.
Overview
Probability
➤ Introduction
➤ Experiments
➤ Events
➤ Probabilities
➤ 5 Simple Rules
➤ Equally Probable
Events
➤ Long Run
➤ Odds
➤ Conditional
Probability
➤ Independence
➤ Dependence
➤ Bayes’ Theorem
Wrapping Up
Lecture #2 - 8/29/2006
✦
These are pretty much ingrained into our mentality, but I
will review them formally.
✦
Although I do not want you to get the picture that Axioms
are all we deal with in this class, these are, sadly, Axioms
of probability.
Slide 9 of 24
Probability Quasi-Axioms
Overview
Probability
➤ Introduction
➤ Experiments
➤ Events
➤ Probabilities
➤ 5 Simple Rules
➤ Equally Probable
Events
➤ Long Run
➤ Odds
➤ Conditional
Probability
➤ Independence
➤ Dependence
➤ Bayes’ Theorem
Wrapping Up
Lecture #2 - 8/29/2006
Given the sample space S and the family of events in S, a
probability function associates with each event A a real
number p(A), the probability of event A, such that the following
axioms are true:
1. p(A) ≥ 0 for every event A.
2. p(S) = 1.0.
3. If there exists some countable set of events,
{A1 , A2 , . . . , AN }, and if these events are all mutually
exclusive,
p(A1 ∪ A2 ∪ . . . ∪ AN ) = p(A1 ) + p(A2 ) + . . . + p(AN )
The probability of the union of mutually exclusive events is the
sum of their separate probabilities.
Slide 10 of 24
Five Simple Rules...
Overview
Probability
➤ Introduction
➤ Experiments
➤ Events
➤ Probabilities
➤ 5 Simple Rules
➤ Equally Probable
Events
➤ Long Run
➤ Odds
➤ Conditional
Probability
➤ Independence
➤ Dependence
➤ Bayes’ Theorem
●
While there aren’t eight simple rules for probability, Hayes
defines five.
●
These rules are really results from the axioms stated on the
previous page.
Wrapping Up
Lecture #2 - 8/29/2006
Slide 11 of 24
Probability Rules
1. p(∽ A) = 1 − p(A)
Overview
Probability
➤ Introduction
➤ Experiments
➤ Events
➤ Probabilities
➤ 5 Simple Rules
➤ Equally Probable
Events
➤ Long Run
➤ Odds
➤ Conditional
Probability
➤ Independence
➤ Dependence
➤ Bayes’ Theorem
2. 0 ≤ p(A) ≤ 1
3. p(∅) = 0 for any S
4. For any two events A and B in S,
p(A ∪ B) = p(A) + p(B) − p(A ∩ B)
5. If the set of events, A, . . . , L constitutes a partition of S, then
p(A ∪ . . . ∪ L) = p(A) + . . . + p(L) = 1.00
Wrapping Up
Lecture #2 - 8/29/2006
Slide 12 of 24
Equally Probable Events
If all elementary events in the sample space S have exactly the
same probability, the probability of any event A is given by
Overview
Probability
➤ Introduction
➤ Experiments
➤ Events
➤ Probabilities
➤ 5 Simple Rules
➤ Equally Probable
Events
➤ Long Run
➤ Odds
➤ Conditional
Probability
➤ Independence
➤ Dependence
➤ Bayes’ Theorem
Wrapping Up
Lecture #2 - 8/29/2006
number of elementary events in A
p(A) =
total number of elementary events in S
n(A)
p(A) =
N
For equally likely elementary events, the probability of an event
A is its relative frequency in the sample space S.
More commonly, the probability will not be equal for all events
in the sample space - then a count of the possible event
outcomes divided by the total number of events will provide the
probability that an even occurs.
Slide 13 of 24
In the Long Run
●
Overview
Probability
➤ Introduction
➤ Experiments
➤ Events
➤ Probabilities
➤ 5 Simple Rules
➤ Equally Probable
Events
➤ Long Run
➤ Odds
➤ Conditional
Probability
➤ Independence
➤ Dependence
➤ Bayes’ Theorem
Wrapping Up
Lecture #2 - 8/29/2006
The probability of an event denotes the relative frequency of
occurrence of that event to be expected in the long run.
✦
The long run relative frequency of occurrence of event A
over trials of the experiment should approach p(A).
●
Bernoulli’s theorem (more or less):
If the probability of occurrence of the event X is p(X)
and if N trials are made, independently and under
exactly the same conditions, the probability that the
relative frequency of occurrence of X differs from p(X)
by any amount, however small, approaches zero as the
number of trials grows indefinitely large.
●
What is not said is that on any trial that an event is certain to
occur.
●
This same theorem underlies many of our sampling routines
we use in studies.
Slide 14 of 24
Odds
●
One common way of expressing the probabilities of two
mutually exclusive events is in terms of betting odds.
●
If the probability of an event is p, then the odds in favor of the
event are p to (1 − p).
●
If the odds in favor of some event are x to y, then the
x
probability of that event is given by p = x+y
.
●
For example, sportsbook.com lists the odds that the San
Francisco 49ers make the 2007 Super Bowl at 175 to 1 (in
Hayes’ system, 1 to 175).
●
So, p(Jon is very happy) = p(49ers make Super Bowl) =
1
1+175 = 0.005682.
●
Conversely, given p = 0.005682, the odds of the 49ers
making it to the Super Bowl are 1−p
p = 175 to 1.
Overview
Probability
➤ Introduction
➤ Experiments
➤ Events
➤ Probabilities
➤ 5 Simple Rules
➤ Equally Probable
Events
➤ Long Run
➤ Odds
➤ Conditional
Probability
➤ Independence
➤ Dependence
➤ Bayes’ Theorem
Wrapping Up
Lecture #2 - 8/29/2006
Slide 15 of 24
Conditional Probability
●
Conditional probabilities are the probability of an event
occurring, given another event has already occurred.
●
For example, we have already talked about the probability
the 49ers made the Super Bowl, how about the conditional
probability of the 49ers making the Super Bowl given the
49ers make the playoffs.
●
We denote the conditional probability of an event B occurring
given event A has already occurred as p(B|A).
Overview
Probability
➤ Introduction
➤ Experiments
➤ Events
➤ Probabilities
➤ 5 Simple Rules
➤ Equally Probable
Events
➤ Long Run
➤ Odds
➤ Conditional
Probability
➤ Independence
➤ Dependence
➤ Bayes’ Theorem
p(B|A) =
p(A ∩ B)
p(A)
Wrapping Up
Lecture #2 - 8/29/2006
Slide 16 of 24
Conditional Probability
●
To use an example from the book, imagine we wanted to
know the probability a student was left handed given the
student was a girl.
●
As a frequency, we could simple count:
Overview
Probability
➤ Introduction
➤ Experiments
➤ Events
➤ Probabilities
➤ 5 Simple Rules
➤ Equally Probable
Events
➤ Long Run
➤ Odds
➤ Conditional
Probability
➤ Independence
➤ Dependence
➤ Bayes’ Theorem
Wrapping Up
# of left-handed girls
p(left-handed|girl) =
total # of girls
●
This is the same as the following:
p(left-handed ∩ girl) = .10
p(girl) = .51
Then,
p(left-handed|girl) =
Lecture #2 - 8/29/2006
.10
= .196
.51
Slide 17 of 24
Independence
●
Knowing about conditional probability can lead us to a
definition of independent events.
●
If two events A and B are independent, then the joint
probability p(A ∩ B) is equal to the probability of A times the
probability of B:
Overview
Probability
➤ Introduction
➤ Experiments
➤ Events
➤ Probabilities
➤ 5 Simple Rules
➤ Equally Probable
Events
➤ Long Run
➤ Odds
➤ Conditional
Probability
➤ Independence
➤ Dependence
➤ Bayes’ Theorem
Wrapping Up
Lecture #2 - 8/29/2006
p(A ∩ B) = p(A)p(B)
●
We can think of independence by looking at conditional
probability.
✦
If p(B|A) = p(B) then because p(B|A) =
be true that p(A ∩ B) = p(A)p(B).
p(A∩B)
p(A)
it must
Slide 18 of 24
Dependence
●
Overview
Probability
➤ Introduction
➤ Experiments
➤ Events
➤ Probabilities
➤ 5 Simple Rules
➤ Equally Probable
Events
➤ Long Run
➤ Odds
➤ Conditional
Probability
➤ Independence
➤ Dependence
➤ Bayes’ Theorem
Conversely, if p(A ∩ B) 6= p(A)p(B) then A and B are said to
be associated, or dependent.
✦
If p(A ∩ B) > p(A)p(B) then A and B are said to be
positively associated.
✦
If p(A ∩ B) < p(A)p(B) then A and B are said to be
negatively associated.
●
Lets have an example...
●
Do you recall the two questions from the background
questionnaire?
Wrapping Up
Lecture #2 - 8/29/2006
Slide 19 of 24
Bayes’ Theorem
●
Overview
Probability
➤ Introduction
➤ Experiments
➤ Events
➤ Probabilities
➤ 5 Simple Rules
➤ Equally Probable
Events
➤ Long Run
➤ Odds
➤ Conditional
Probability
➤ Independence
➤ Dependence
➤ Bayes’ Theorem
When considering conditional probabilities, Bayes’ Theorem
usually isn’t too far behind:
p(A|B) =
p(B|A)p(A)
p(B|A)p(A) + p(B| ∽ A)p(∽ A)
●
This theorem is generally useless for most of the things we
will learn in this course, but really sets the foundation for
many Bayesian statistical methods used in advanced
statistics.
●
A classic example of Bayes’ Theorem comes from the
problems encountered with disease diagnosis.
Wrapping Up
Lecture #2 - 8/29/2006
Slide 20 of 24
Bayes’ Theorem Example
●
Let’s imagine you get so
caught up in all this talk
about probability, odds, and
gambles that you end up
spending a lot of time in my
favorite city, Las Vegas.
●
You end up thinking you
have a problem, so you
take a survey which
indicates you meet the
DSM definition for being a
pathological gambler.
●
What are the chances you actually are a pathological
gambler?
Overview
Probability
➤ Introduction
➤ Experiments
➤ Events
➤ Probabilities
➤ 5 Simple Rules
➤ Equally Probable
Events
➤ Long Run
➤ Odds
➤ Conditional
Probability
➤ Independence
➤ Dependence
➤ Bayes’ Theorem
Wrapping Up
Lecture #2 - 8/29/2006
Slide 21 of 24
Bayes’ Theorem Example
●
Overview
Probability
➤ Introduction
➤ Experiments
➤ Events
➤ Probabilities
➤ 5 Simple Rules
➤ Equally Probable
Events
➤ Long Run
➤ Odds
➤ Conditional
Probability
➤ Independence
➤ Dependence
➤ Bayes’ Theorem
Wrapping Up
Lecture #2 - 8/29/2006
To figure this problem out, lets define event A as truly being
a pathological gambler.
✦
The DSM states that p(A) = 0.03.
●
We will define event B as having the test result indicate you
are a pathological gambler.
●
We know that if a person is a pathological gambler, the test is
positive 80% of the time (or p(B|A) = 0.80).
●
We also know that if a person is not a pathological gambler,
the test is negative 90% of the time (or p(∽ B| ∽ A) = 0.95).
p(A|B) =
p(B|A)p(A)
=?
p(B|A)p(A) + p(B| ∽ A)p(∽ A)
Slide 22 of 24
Final Thought
●
Today’s class, while
seeming trivial, lays the
foundation for the statistical
tests you will learn about
and use throughout the
remainder of the year.
●
The simple experiments we
talked about today can be
generalized to what we do
when we conduct research.
●
The notions of conditional probability and independence play
a large role when considering the types of data we collect.
●
Bayes theorem sets the foundation for many statistical
treatments that are helpful in practice.
Overview
Probability
Wrapping Up
➤ Final Thought
➤ Next Class
Lecture #2 - 8/29/2006
Slide 23 of 24
Next Time
Overview
●
Random variables (Hayes, Chapter 2.13-2.22).
●
Graphical displays of statistical distributions.
●
More good clean stats fun.
Probability
Wrapping Up
➤ Final Thought
➤ Next Class
Lecture #2 - 8/29/2006
Slide 24 of 24