Download 2 Basics of Probability and Statistics

2
Basics of Probability and Statistics
2.1
Sample Space, Events, and Probability Measure
1. Random Experiment: A random experiment is a process leading to at least two possible outcomes with uncertainty as to which will occur. E.G. toss a coin, throw a dice,
pull the lever of a slot machine etc.
2. Sample Space (S): A sample space is the set of all possible outcomes of an experiment.
• Toss a coin: S = {H, T } .
• Toss two coins: S = {HH, HT, T H, T T } .
• NY Mets play a doubleheader: S = {W W, W L, LW, LL} .
Each element of the sample space is called a sample point.
3. Events: An event is a subset of the sample space. For example, list all the subsets of
S = {H, T } : ∅, {H} , {T } , {H, T } . These are the four possible events from tossing a
coin, where ∅ is called null event (it is there for mathematical completeness), it means
that nothing happens. {H} is the event of head; {T } is the event of tail; {H, T } is
the event of “anything happens”.
• Exercise: In the “toss two coins” example, express the event “one head and one
tail”. {HT, T H} .
• Exercise: In the “toss two coins” example, express the event “the first coin is a
head”.
4. The collection of all the events is called the σ-algebra, denoted by S. Example, the
σ-algebra for “toss a coin” example is S = {∅, {H} , {T } , {H, T }} . The σ-algebra for
“toss two coins” example is
S = {∅, {HH} , {HT } , {T H} , {T T } , {HH, HT } , {HH, T H} ,
{HH, T T } , {HT, T H} , {HT, T T } , {T H, T T } , {HH, HT, T H} ,
{HH, T H, T T } , {HH, HT, T T } , {HT, T H, T T } , {HH, HT, T H, T T }}
5. Events are said to be mutually exclusive if the occurrences of one event prevents the
occurrences of another event at the same time. In the “toss two coins” example,
event “one head one tail” {HT, T H} and the event “two heads” {HH} are mutually
exclusive.
6
6. A probability measure P is a mapping from a σ-algebra to [0, 1] that satisfies:
• P (∅) = 0, P (S) = 1;
• P (A) ≥ 0 for all A ∈ S;
• If A1 ∈ S, A2 ∈ S and A1 and A2 are mutually exclusive, then P (A1 ∪ A1 ) =
P (A1 ) + P (A2 ) .
Notation: We use A1 + A2 (or A1 ∪ A2 ) to denote the occurrences of either A1 or A2 ;
use A1 A2 (or A1 ∩ A2 ) to denote the simultaneous occurrences of events A1 and A2 .
7. If A, B are any events, they are said to be statistically independent events if the
probability of their occurring together is equal to the product of their individual
probabilities, that is,
P (A ∩ B) ≡ P (AB) = P (A) P (B) .
• Example: “toss two fair coins”. What is the probability of obtaining a head
on the first coin and a head on the second coin? Event A is the event of head
on the first coin {HH, HT }; event B is the event of head on the second coin
{HH, T H} . The event AB is {HH} . Common sense that A and B are statisti-
cally independent, hence P ({HH}) = 1/4.
8. If events A and B are not mutually exclusive, then [illustrate by a diagram]
P (A + B) = P (A) + P (B) − P (AB)
where P (A + B) is the probability that either A or B occurs, P (AB) is the probability
that A and B both occurs. If A and B are mutually exclusive, of course P (AB) =
P (A ∩ B) = P (∅) = 0 by the definition of probability measure.
• Example: A card is drawn from a deck of cards [52 total cards] What is the
probability that it will be either a heart or a queen. Event A : the card is a
heart; event B : the card is a queen.
P (A + B) = P (A) + P (B) − P (AB)
= 13/52 + 4/52 − 1/52 = 4/13.
7
9. Conditional probability: This concept answers the following question: what is the
probability that event A occurs, knowing that event B has occurred (this implies that
P (B) > 0). This is called the conditional probability of event A, conditional on event
B occurring, and it is denoted by P (A|B) , The formula is:
P (A|B) =
P (AB)
P (B)
where the numerator P (AB) is called the joint probability of events A and B and the
denominator is called the marginal probability of event B.
• Example 1: I randomly draw a card from a deck of cards. Suppose that I tell you
the card drawn is a queen, what is the probability that card is a heart? Event
B : the card is a queen; event A : the card is a heart. The probability is
P (A|B) =
P (AB)
1/52
=
= 1/4.
P (B)
4/52
• Example 2: In Economics 101 there are a total of 500 students, of which 300 are
males, 200 are females. We know that 100 males and 60 females plan to major
in economics. A student is randomly selected from the class, and it is found
that this student plan to major in economics. What is the probability that this
student is a male? Answer: Event A : the student is a male; Event B : the
student is an economics major.
Pr (A|B) =
P (AB)
100/500
=
= 5/8.
P (B)
160/500
What is the unconditional probability that a randomly drawn student is a male?
P (A) = 300/500 = 0.6.
• Prove: If events A and B are statistically independent, then P (A|B) = P (A) .
[interpretation: if two events are statistically independent, then knowing one
event has occurred does not reveal any information about the likelihood of the
other event.]
10. Law of Total Probability: Let the sample space S be partitioned into k mutually exclusive and exhaustive events C1 , ..., Ck such that P (Ci ) > 0 for all i. Let C be another
event such that P (C) > 0. We have
C = C ∩ S = C ∩ (C1 ∪ C2 ∪ ... ∪ Ck )
= (C ∩ C1 ) ∪ (C ∩ C2 ) ∪ ... ∪ (C ∩ Ck )
8
hence
P (C) = P (C ∩ C1 ) + ... + P (C ∩ Ck )
= P (C1 ) P (C|C1 ) + ... + P (Ck ) P (C|Ck )
k
X
P (Ci ) P (C|Ci ) .
=
i=1
11. Bayes’ Theorem:
P (Cj |C) =
=
P (C ∩ Cj )
P (C)
P (C|Cj ) P (Cj )
Pk
i=1 P (Ci ) P (C|Ci )
where Ci , i = 1, ..., k are mutually exclusive and exhausive events of the sample space
S. P (Ci ) are called prior probabilities of event Ci and P (Ci |C) are known as the
posterior probabilities of event Ci after it is known that event C occured.
9