Download Probability Review 2

PROBABILITY NOTES - PR2
RANDOM VARIABLES AND PROBABILITY DISTRIBUTIONS
Random variable ? : A random variable is a function on a sample space : . This function
assigns a real number ?² ³ to each sample point  : . Often a random variable is simply equal
to the sample point , if the sample points are numerical values - for example, the sample space
representing the number of spots that turn up when an ordinary die is tossed is
: ~ ¸Á Á Á Á Á ¹, and ?² ³ ~ describes the random variable ? which is the number of
spots that turn up. Alternatively, suppose that a gamble based on the outcome of the toss of a die
pays $10 if an even number is tossed, and pays $20 if an odd number is tossed. The payoff can
be represented by the random variable @ , where @ ² ³ ~ if is even, and @ ² ³ ~ if is
odd. A random variable is sometimes described in terms of the outcome of a random experiment
(such as tossing a die), or may be described without explicit reference to the underlying random
experiment or sample space (such as the prime rate of interest two years from now). Given a set
of real numbers, (, then 7 ´?  (µ is defined to be the probability of the event represented by
the related subset of the sample space 7 ¸ ¢ ?² ³  (¹ . Using random variable @ from the
$10 for even, $20 for odd die toss example, we have, as an example,
7 ´@ ‚ µ ~ 7 ´¸ ¢ @ ² ³ ‚ µ ~ 7 ´¸Á Á ¹µ , since these are the sample points for which
@ ² ³ ‚ (for a fair die, this probability is ).
Discrete random variable: The random variable ? is discrete and is said to have a discrete
distribution if it can take on values only from a finite or countable infinite sequence (usually the
integers or some subset of the integers). As an example, consider the following two random
variables related to successive tosses of a coin? ~ if the first head occurs on an even-numbered toss, ? ~ if the first head occurs on an
odd-numbered toss;
@ ~ , where is the number of the toss on which the first head occurs.
Both ? and @ are discrete random variables, where ? can take on only the values or , and @
can take on any non-negative integer value. Both ? and @ are based on the same sample space the sample points are sequences of tail coin flips ending with a head coin flip:
: ~ ¸/ Á ; / Á ; ; / Á ; ; ; / Á ; ; ; ; / Á ÀÀÀ¹ . Then,
?²/³ ~ (a head on flip one, an odd-numbered flip), ?²; /³ ~ Á ?²; ; /³ ~ , ...
@ ²/³ ~ (first head on flip ), @ ²; /³ ~ Á @ ²; ; /³ ~ Á @ ²; ; ; /³ ~ Á ÀÀÀ
1
Probability function of a discrete random variable: The probability function (p.f.) of a
discrete random variable is usually denoted ²%³Á ? ²%³, ²%³ or % , and is equal to 7 ´? ~ %µ.
The probability function must satisfy (i)  ²%³  for all % , and (ii) ²%³ ~ .
%
Given a set ( of real numbers, 7 ´?  (µ ~ ²%³ .
%(
Continuous random variable: A continuous random variable usually can assume numerical
values from an interval of real numbers, perhaps the whole set of real numbers l. As an
example, the length of time between successive streetcar arrivals at a particular (in service)
streetcar stop could be regarded as a continuous random variable (assuming that time
measurement can be made perfectly accurate).
Probability density function: A continuous random variable ? usually has a probability
density function (p.d.f.) denoted ²%³ or ? ²%³ (or sometimes denoted ²%³ ), which is a
continuous function except possibly at a finite number of points. Probabilities related to ? are
found by integrating the density function 7 ´?  ²Á ³µ ~ 7 ´  ?  µ is defined to be equal to ²%³ % .
B
²%³ must satisfy (i) ²%³ ‚ for all % , and (ii) ²%³ % ~ .
cB
Often, the region of non-zero density is finite, and ²%³ ~ outside that interval.
If ²%³ is continuous except at a finite number of points, then probabilities are defined and
calculated as if ²%³ was continuous everywhere (the discontinuities are ignored).
For example, suppose that ? has density function ²%³ ~ F
% for %
, elsewhere
. Then
B
satisfies the requirements for a density function, since cB ²%³ % ~ % % ~ .
À
À
Then, for example 7 ´À  ?  Àµ ~ À % % ~ % e ~ À .
À
Mixed distribution: A random variable that has some points with non-zero probability mass,
and with a continuous p.d.f. elsewhere is said to have a mixed distribution. The sum of the
probabilities at the discrete points of probability plus the integral of the density function on the
continuous region for ? must be 1.
For example, suppose that ? has probability of À at ? ~ , and ? is a continuous random
variable on the interval ²Á ³ with density function ²%³ ~ % for  %  , and ? has no
density or probability elsewhere. This satisfies the requirements for a random variable since
7 ´? ~ µ b ²%³ % ~ À b % % ~ À b À ~ .
À
Then, 7 ´  ?  Àµ ~ % % ~ ÀÁ and
7 ´  ?  Àµ ~ 7 ´? ~ µ b 7 ´  ?  Àµ ~ À b À ~ À
.
2
Cumulative distribution function (and survival function): Given a random variable ? , the
cumulative distribution function of ? (also called the distribution function, or c.d.f.) is
- ²%³ ~ 7 ´?  %µ (also denoted -? ²%³ ). The survival function is the complement of the
distribution function, :²%³ ~ c - ²%³ ~ 7 ´? € %µ . The event ? € % is referred to as a
"tail" of the distribution.
For a discrete random variable with probability function ²%³, - ²%³ ~ ²$³ , and
$%
in this case - ²%³ is a "step function" - it has a jump (or step increase) at each point with non-zero
probability, while remaining constant until the next jump.
%
If ? has a continuous distribution with density function ²%³, then - ²%³ ~ cB ²!³ !
and - ²%³ is a continuous, differentiable, non-decreasing function such that
Z
% - ²%³ ~ - ²%³ ~ ²%³ . If ? has a mixed distribution, then - ²%³ is continuous except at the
points of non-zero probability mass, where - ²%³ will have a jump. For any c.d.f.
7 ´  ?  µ ~ - ²³ c - ²³ Á lim - ²%³ ~ Á lim - ²%³ ~ .
%¦B
%¦cB
Examples of distribution functions:
? ~ number turning up when tossing one fair die, so ? has probability function
? ²%³ ~ 7 ´? ~ %µ ~ for % ~ Á Á Á Á Á . ? is a discrete random variable.
if %  if  %  -? ²%³ ~ 7 ´?  %µ ~ J
if  %  if  %  if  %  if  %  if % ‚ @ is a continuous random variable on the interval ²Á ³ with density function
@ ²&³ ~ F
& for %
, elsewhere
.
if &  Then -@ ²&³ ~ H & if  &  if & ‚ 3
A has a mixed distribution on the interval ´Á ³. A has probability of À at A ~ , and A has
density function A ²'³ ~ ' for  '  , and A has no density or probability elsewhere.
if '  À if ' ~ Then, -A ²'³ ~ J
.
À b ' if  '  if ' ‚ Some results and formulas relating to this section:
(i) For a continuous random variable ? ,
7 ´  ?  µ ~ 7 ´  ?  µ ~ 7 ´  ?  µ ~ 7 ´  ?  µ , so that when
calculating the probability for a continuous random variable on an interval, it is irrelevant
whether or not the endpoints are included. For a continuous random variable, 7 ´? ~ µ ~ ;
non-zero probabilities only exist over an interval, not at a single point. Also, for a continuous
c ²%³ ~ ´ c - ²%³µ .
random variable, the hazard rate or failure rate is ²%³ ~ c-²%³
%
Z
(ii) If ? has a mixed distribution, then 7 ´? ~ !µ will be non-zero for some value(s) of !, and
7 ´  ?  µ will not always be equal to 7 ´  ?  µ (they will not be equal if ? has a nonzero probability mass at either or ).
(iii) ²%³ may be defined piecewise, meaning that ²%³ is defined by a different algebraic
formula on different intervals.
(iv) A continuous random variable may have two or more different, but equivalent p.d.f.'s, but
the difference in the p.d.f.'s would only occur at a finite (or countably infinite) number of points.
The c.d.f. of a random variable of any type is always unique to that random variable.
Example 16: A die is loaded in such a way that the probability of the face with dots turning up
is proportional to for ~ Á Á Á Á Á . What is the probability, in one roll of the die, that an
even number of dots will turn up?
Solution: Let ? denote the random variable representing the number of dots that appears when
the die is rolled once. Then, 7 ´? ~ µ ~ 9 h for ~ Á Á Á Á Á , where 9 is the
proportional constant. Since the sum of all of the probabilities of points in that can occur must
be . it follows that 9 h ´ b b b b b µ ~ , so that 9 ~ .
Then, 7 ´even number of dots turns upµ ~ 7 ´µ b 7 ´µ b 7 ´
µ ~ bb
~ .
4
U
Example 17: An ordinary single die is tossed repeatedly until the first even number turns up.
The random variable ? is defined to be the number of the toss on which the first even number
turns up. Find the probability that ? is an even number.
Solution: ? is a discrete random variable that can take on an integer value of or more. The
probability function for ? is ²%³ ~ 7 ´? ~ %µ ~ ² ³% (this is the probability of % c successive odd tosses followed by an even toss - the same as in Example 92 earlier in these
notes). Then, 7 ´? is
² ³
evenµ ~ 7 ´µ b 7 ´µ b 7 ´
µ b Ä ~ ² ³ b ² ³ b ² ³
b Ä ~ c² ³ ~ . U
Example 18: The continuous random variable ? has density function
²%³ ~ c % for c À  %  À (and ²%³ ~ elsewhere). Find 7 ´  ?  µ.
Solution: 7 ´À  ?  Àµ ~ 7 ´À  ?  Àµ , since there is no density for ? at
À
points greater than À. The probability is À ² c % ³ % ~ .
U
Example 19: Suppose that the continuous random variable ? has the cumulative distribution
function - ²%³ ~ b
c% for c B  %  B . Find ? 's density function.
Solution: The density function for a continuous random variable is the first derivative of the
c%
cumulative distribution function. The density function of ? is ²%³ ~ - Z ²%³ ~ ²b
c% ³ . U
Example 20: ? is a random variable for which 7 ´?  %µ ~ c c% for % ‚ , and
7 ´?  %µ ~ for %  . Which of the following statements is true?
A) 7 ´? ~ µ ~ c c and 7 ´? ~ µ ~ c c
B) 7 ´? ~ µ ~ c c and 7 ´?  µ ~ c c
C) 7 ´? ~ µ ~ c c and 7 ´?  µ ~ c c
D) 7 ´?  µ ~ c c and 7 ´?  µ ~ c c
E) 7 ´?  µ ~ c c and 7 ´? ~ µ ~ c c
Solution: Since 7 ´?  %µ ~ c c% for % ‚ , it follows that 7 ´?  µ ~ c c .
But 7 ´?  %µ ~ if %  , and thus 7 ´?  µ ~ , so that 7 ´? ~ µ ~ c c
(since 7 ´?  µ ~ 7 ´?  µ b 7 ´? ~ µ ). This eliminates answers C and D. Since
the distribution function for ? is continuous (and differentiable) for % € , it follows that
7 ´? ~ %µ ~ for % € . This eliminates answers A, B and C.
This is an example of a random variable ? with a mixed distribution - a point of probability at
? ~ , and a continuous distribution for ? € .
U
5
Example 21: A continuous random variable ? has the density function
%  %  ²%³ ~ H c%  %  . Find 7 ´À  ?  Àµ .
Á elsewhere
À
À
À
Solution: 7 ´À  ?  Àµ ~ À ²%³ % ~ À % % b À c%
% ~ .
Note that since ? is a continuous random variable, the probability 7 ´À  ?  Àµ
would be the same as 7 ´À  ?  Àµ . This is an example of a density function defined
piecewise. Also, note that if the density function was defined to be
%  %  ²%³ ~ H % ~ °
( density at % ~ ), then all probabilities are unchanged (since the
c% %
two density functions and differ at one point, probability calculations, which are based on
integrals of the density function over an interval, are the same for both and ). U
EXPECTATION AND OTHER DISTRIBUTION PARAMETERS
Expected value of a random variable: For a random variable ? , the expected value is denoted
,´?µ , or ? or .
For a discrete random variable, the expected value of ? is % h ²%³ , where the sum is taken
over all points % at which ? has non-zero probability. For instance, if ? is the result of one toss
of a fair die, then ,´?µ ~ h b h b Ä b h ~ À
B
For a continuous random variable, the expected value is cB % h ²%³ % - although the integral
is written with lower limit c B and upper limit B, the interval of integration is the interval of
non-zero-density for ? .
Note that is the probability function in the discrete case, and is the density function in the
continuous case. The expected value of ? is also called the expectation of ? , or the mean of
?. The expected value is the "average" over the range of values that ? can be, or the "center" of
the distribution.
Expectation of ²%³: If is a function, then ,´²?³µ is equal to ²%³ h ²%³
%
B
if ? is a discrete random variable, and it is equal to cB ²%³ h ²%³ % if ? is a
continuous random variable.
6
Moments of a random variable: If ‚ is an integer, then the -th moment of ? is ,´? µ.
If the mean of ? is , then the -th central moment of ? (about the mean ) is ,´²? c ³ µ.
or . It is defined to be
Variance of ? : The variance of ? is denoted = ´?µ , = ´?µ , ?
~ ,´? µ c ²,´?µ³
equal to = ´?µ ~ ,´²?c? ³ µ ~ ,´? µ c ?
(the variance is the 2nd central moment of ? about its mean). The variance is a measure of the
"dispersion" of ? about the mean - a large variance indicates significant levels of probability or
density for points far from ,´?µ. The variance is always ‚ (the variance of ? is equal to only if ? has a discrete distribution with a single point and probability at that point (not
random at all).
Standard deviation of ? : The standard deviation of the random variable ? is the square root
of the variance, and is denoted ? ~ l= ´?µ .
Moment generating function of random variable ? : The moment generating function of ?
(m.g.f.) is denoted 4? ²!³ Á ? ²!³ Á 4 ²!³ or ²!³, and it is defined to be
B
4? ²!³ ~ ,´!? µ , which is either !% ²%³ or cB !% ²%³ % if ? is discrete or
%
continuous, respectively. It is always true that 4? ²³ ~ . The moment generating function of
? might not exist for all real numbers, but usually exists on some interval of real numbers. The
function ´4? ²!³µ is called the cumulant generating function.
Percentiles of a distribution: If   , then the -th percentile of the distribution of
? is the number which satisfies both of the following inequalities:
7 ´?  µ ‚ and 7 ´? ‚ µ ‚ c . For a continuous random variable, it is sufficient to
find the for which 7 ´?  µ ~ . If ~ À , the 50-th percentile of a
distribution is referred to as the median of the distribution - it is the point 4 for which
7 ´?  4 µ ~ À . The median 4 is the 50% probability point - half of the distribution
probability is to the left of 4 and half is to the right.
The mode of a distribution: The mode is any point at which the probability or density
function ²%³ is maximized.
The skewness of a distribution: If the mean of random variable ? is and the variance is then the skewness is defined to be ,´²? c ³ µ° .
7
Some results and formulas relating to this section:
(i) The mean of a random variable ? might not exist, it might be b B or c B, and the
variance of ? might be b B. For example, the continuous random variable ? with
p.d.f. ²%³ ~ F %
for %‚
, otherwise
B
has expected value % h % % ~ b B .
(ii) For any constants Á and and functions and ,
,´ ²?³ b ²?³ b µ ~ ,´ ²?³µ b ,´ ²?³µ b (iii) If ? is a random variable defined on the interval ´Á B) ( ²%³ ~ for %  ),
B
then ,´?µ ~ b ´ c - ²%³µ % , and if ? is defined on the interval ´Á µ , where
 B, then ,´?µ ~ b ´ c - ²%³µ % . This relationship is valid for any random
variable, discrete, continuous or with a mixed distribution. As a special, case, if ? is a
non-negative random variable (defined on ´Á B³ or ²Á B³ ) then
B
,´?µ ~ ´ c - ²%³µ %
(iv) Jensen's inequality: If is a function and ? is a random variable such that
%
²%³ ~ ZZ ²%³ ‚ at all points % with non-zero density or probability for ? ,
then ,´²?³µ ‚ ²,´?µ³ , and if ZZ € then ,´²?³µ € ²,´?µ³À The inequality
reverses if ZZ  . For example, if ²%³ ~ % , then ZZ ²%³ ~ ‚ for any %, so that
,´? µ ‚ ²,´?µ³ (this is also true since = ´?µ ~ ,´? µ c ²,´?µ³ ‚ for any random
variable ? ). As another example, if ? is a positive random variable (i.e., ? has non-zero
c
density or probability only for % ‚ ), and ²%³ ~ l% , then ZZ ²%³ ~ %
°  for % € ,
and it follows from Jensen's inequality that ,´l?µ  l,´?µ .
(v) If and are constants, then = ´? b µ ~ = ´?µ .
(vi) Chebyshev's inequality: If ? is a random variable with mean ? and standard
deviation ? , then for any real number € , 7 ´O? c ? O € ? µ  .
(vii) Suppose that for the random variable ? , the moment generating function 4? ²!³
exists in an interval containing the point ! ~ . Then
²³
! 4? ²!³e!~ ~ 4? ²³ ~ ,´? µ ,
4?Z ²³
´4
²!³µ
~
~ ,´?µ ,
e
?
!
!~ 4? ²³
the -th moment of ? , and
and !
´4? ²!³µe
!~
~ = ´?µ .
The Taylor series expansion of 4? ²!³ expanded about the point ! ~ is
B !
4? ²!³ ~ [
,´? µ ~ b ! h ,´?µ b ! h ,´? µ b !
h ,´? µ b Ä
~
8
If ? and ? are random variables, and 4? ²!³ ~ 4? ²!³ for all values of ! in an
interval containing ! ~ , then ? and ? have identical probability distributions.
(viii) The median (50th percentile) and other percentiles of a distribution are not always
unique. For example, if ? is the discrete random variable with probability function
²%³ ~ À for % ~ Á Á Á , then the median of ? would be any point from to ,
but the usual convention is to set the median to be the midpoint between the two
"middle" values of ? , 4 ~ À .
(ix) The distribution of the random variable ? is said to be symmetric about the point
if ² b !³ ~ ² c !³ for any value of !. It follows that the expected value of ?
and the median of ? is . Also, for a symmetric distribution, any odd-order central
moments about the mean are , i.e. ,´²? c ³ µ ~ if ‚ is an odd integer.
(x) If ,´?µ ~ , = ´?µ ~ and A is defined to be A ~
,´Aµ ~ and = ´Aµ ~ .
?c
, then
Example 22: Let ? equal the number of tosses of a fair die until the first "1" appears.
Find ,´?µ.
Solution: ? is a discrete random variable that can take on an integer value ‚ . The
probability that the first 1 appears on the %-th toss is ²%³ ~ ² ³%c ² ³ for % ‚ (% c tosses that are not followed by a 1). This is the probability function of ? . Then
B
B
~
~
,´?µ ~ h ²³ ~ h ² ³c ² ³ ~ ² ³´ b ² ³ b ² ³ b ĵ .
We use the general increasing geometric series relation b b b Ä ~ ²c³
,
so that ,´?µ ~ ² ³ h ~ . U
²c ³
Example 23: Given that the density function of ? is ²% ¢ ³ ~ c% , for % € , and elsewhere, find the -th moment of ? , where is a non-negative integer (assuming that € ).
B
Solution: The -th moment of ? is ,´? µ ~ % h c% % . Applying integration by
parts, this can be written as
%~B
B
B
B % ² c c% ³ ~ c % c% e
c c %c c% % ~ %c c% % À
%~
Repeatedly applying integration by parts results in ,´? µ ~ . It is worthwhile noting the
general form of the integral that appears in this example - if ‚ is an integer and € , then
B
by repeated applications of integration by parts, we have ! c! ! ~ [
b , so that in this
B
B
[
example % c% % ~ % c% % ~ h [
U
b ~ .
[
9
Example 24: A fair die is tossed until the first 1 appears. Let % equal the number of tosses
required, % ~ Á Á Á ÀÀÀ You are to receive ²À³% dollars if the appears on the %-th toss. What
is the expected amount that you will receive?
Solution: This is the same distribution as in Example 1 above, with the probability that the
first 1 appears on the %-th toss being ² ³%c ² ³ for % ‚ (% c tosses that are not followed
by a 1), and the amount received in that case is ²%³ ~ ²À³% . Then, the expected amount
B
received is ,´²?³µ ~ ,´²À³? µ ~ ²À³ h ² ³c ² ³ ~ ² ³´ b ² ³ b ² ³ b ĵ ~ .U
~
Example 25: A continuous random variable ? has density function
²%³ ~ F
cO%O if O%O
, elsewhere
. Find = ´?µ.
Solution: The density of ? is symmetric about (since ²%³ ~ ² c %³), so that
,´?µ ~ (this can be verified directlyÂ
,´?µ ~ c %² c O%O³ % ~ c %² b %³ % b %² c %³ % ~ c b ~ ).
Then, = ´?µ ~ ,´? µ c ²,´?µ³ ~ ,´? µ ~ % ² c O%O³ %
c
~ c % ² b %³ % b % ² c %³ % ~ .
U
Example 26: The moment generating function of ? is c! for !  , where € .
Find = ´?µ.
e ~ ,
Solution: = ´?µ ~ ,´? µ c ²,´?µ³ . ,´?µ ~ 4?Z ²³ ~ ²c!³
!~ and ,´? µ ~ 4?ZZ ²³ ~ ²c!³ e ~ S = ´?µ ~ c ² ³ ~ .
!~
Alternatively, 4? ²!³ ~ ² c! ³ ~ c ² c !³ S !
´4? ²!³µ ~ c!
and !
~ À
´4? ²!³µ ~ ²c!³ so that = ´?µ ~ ! ´4? ²!³µe
!~
U
Example 27: The continuous random variable ? has p.d.f. ²%³ ~ h cO%O for
c B  %  B . Find the 87.5-th percentile of the distribution.
Solution: The 87.5-th percentile is the number for which
À ~ 7 ´?  µ ~ cB ²%³ % ~ cB h cO%O % .
Note that this distribution is symmetric about , since ² c %³ ~ ²%³, so the mean and median
are both . Thus, € , and so
h cO%O % ~ h cO%O % b h cO%O % ~ À b h c% %
cB cB ~ À b ² c c ³ ~ À S ~ c ²À³ ~ .
10
U