Download Chapter 3: Random Variables1

Chapter 3: Random Variables1
Yunghsiang S. Han
Graduate Institute of Communication Engineering,
National Taipei University
Taiwan
E-mail: [email protected]
1 Modified
from the lecture notes by Prof. Mao-Ching Chiu
Y. S. Han
Random Variables
3.1 The Notion of a Random Variable
• A random variable X is a function that
assigns a real number, X(ζ), to each outcome
ζ in the sample space.
Graduate Institute of Communication Engineering, National Taipei University
1
Y. S. Han
Random Variables
2
Example:
• Toss a coin three times.
•
S = {HHH, HHT, HTH, THH, HTT,
THT, TTH, TTT}.
• Let X be the number of heads.
ζ
HHH
HHT
HTH
THH
HTT
THT
TTH
TTT
X(ζ)
3
2
2
2
1
1
1
0
• X is a random variable with SX = {0, 1, 2, 3}.
Graduate Institute of Communication Engineering, National Taipei University
Y. S. Han
Random Variables
• S: Sample space of a random experiment
• X: Random variable X : S → SX
• SX is a new sample space
• Let B ⊆ SX and A = {ζ : X(ζ) ∈ B}. Then
P [B] = P [A] = P [{ζ : X(ζ) ∈ B}]
• A and B are equivalent events.
Graduate Institute of Communication Engineering, National Taipei University
3
Y. S. Han
Random Variables
Graduate Institute of Communication Engineering, National Taipei University
4
Y. S. Han
Random Variables
3.2 Cumulative Distribution Function
• Cumulative distribution function (cdf)
FX (x) = P [X ≤ x] for − ∞ < x < ∞
• In underlying sample space
FX (x) = P [{ζ : X(ζ) ≤ x}]
• FX (x) is a function of the variable x.
Graduate Institute of Communication Engineering, National Taipei University
5
Y. S. Han
Random Variables
Properties of cdf
1. 0 ≤ FX (x) ≤ 1.
2. limx→∞ FX (x) = 1.
3. limx→−∞ FX (x) = 0.
4. If a < b, then FX (a) ≤ FX (b).
5. FX (x) is continuous from the right, i.e., for h > 0
FX (b) = lim FX (b + h) = FX (b+ ).
h→0
6. P [a < X ≤ b] = FX (b) − FX (a), since
{X ≤ a} ∪ {a < X ≤ b} = {X ≤ b}.
Graduate Institute of Communication Engineering, National Taipei University
6
Y. S. Han
Random Variables
7. P [X = b] = FX (b) − FX (b− ).
8. P [X > x] = 1 − FX (x).
Graduate Institute of Communication Engineering, National Taipei University
7
Y. S. Han
Random Variables
• X: the number of heads in three tosses of a fair coin
• Let δ be a small positive number. Then
FX (1 − δ) = P [X ≤ 1 − δ] = P {0 heads} = 1/8,
FX (1) = P [X ≤ 1] = P [0 or 1 heads] = 1/8+3/8 = 1/2,
FX (1 + δ) = P [X ≤ 1 + δ] = P [0 or 1 heads] = 1/2.
• Write in unit step function
(
u(x) =
0, x < 0
1, x ≥ 0,
3
3
1
1
FX (x) = u(x) + u(x − 1) + u(x − 2) + u(x − 3).
8
8
8
8
Graduate Institute of Communication Engineering, National Taipei University
8
Y. S. Han
Random Variables
Graduate Institute of Communication Engineering, National Taipei University
9
Y. S. Han
Random Variables
10
Example: The transmission time X of messages in a
communication system obeys the exponential probability
law with parameter λ, i.e.,
P [X > x] = e−λx
x>0
Find the cdf of X and P [T < X ≤ 2T ], where T = 1/λ.
FX (x) = P [X ≤ x] = 1 − P [X > x] =
(
0,
x<0
1 − e−λx , x ≥ 0.
P [T < X ≤ 2T ] = 1 − e−2 − (1 − e−1 ) = e−1 − e−2 .
Graduate Institute of Communication Engineering, National Taipei University
Y. S. Han
Random Variables
Graduate Institute of Communication Engineering, National Taipei University
11
Y. S. Han
Random Variables
Types of random variables
• Discrete random variable, SX = x0 , x1 , . . .
X
FX (x) =
pX (xk )u(x − xk ),
k
where xk ∈ SX and pX (xk ) = P [X = xk ] is the
probability mass function (pmf) of X.
• Continuous random variable
Z
FX (x) =
x
f (t)dt
−∞
• Random variable of mixed type
FX (x) = pF1 (x) + (1 − p)F2 (x)
Graduate Institute of Communication Engineering, National Taipei University
12
Y. S. Han
Random Variables
3.3 Probability Density Function
• Probability density function (pdf) of X is
dFX (x)
.
fX (x) =
dx
Graduate Institute of Communication Engineering, National Taipei University
13
Y. S. Han
Random Variables
Graduate Institute of Communication Engineering, National Taipei University
14
Y. S. Han
Random Variables
Properties of pdf
1. fX (x) ≥ 0.
Rb
2. P [a ≤ X ≤ b] = a fX (x)dx.
Rx
3. FX (x) = −∞ fX (t)dt.
R∞
4. −∞ fX (t)dt = 1.
Graduate Institute of Communication Engineering, National Taipei University
15
Y. S. Han
Random Variables
The pdf of the uniform random variable
(
1
, a≤x≤b
b−a
fX (x) =
0,
otherwise
FX (x) =



 0,
x−a
,
b−a


 1,
x<a
a≤x≤b
x>b
Graduate Institute of Communication Engineering, National Taipei University
16
Y. S. Han
Random Variables
Graduate Institute of Communication Engineering, National Taipei University
17
Y. S. Han
Random Variables
18
Pdf for discontinuous cdf
• Unit step function
u(x) =
(
0 x<0
1 x≥0
• Delta function δ(t)
u(x) =
Z
x
δ(t)dt
−∞
• Cdf for a discrete random variable
Z
X
FX (x) =
pX (xk )u(x − xk ) =
k
x
fX (t)dt
−∞
Graduate Institute of Communication Engineering, National Taipei University
Y. S. Han
Random Variables
→ fX (x) =
P
k
pX (xk )δ(x − xk )
Graduate Institute of Communication Engineering, National Taipei University
19
Y. S. Han
Random Variables
20
Conditional cdf ’s and pdf ’s
• Conditional cdf of X given A
P [{X ≤ x} ∩ A]
FX (x|A) =
P [A]
if P [A] > 0
• Conditional pdf of X given A
d
fX (x|A) =
FX (x|A)
dx
Graduate Institute of Communication Engineering, National Taipei University
Y. S. Han
Random Variables
3.4 Some Important Random Variables
• Bernoulli random variable: Let A be an event. The
indicator function for A is
(
0 ζ∈
/A
IA (ζ) =
.
1 ζ∈A
IA is the Bernoulli random variable. Ex: toss a coin.
• Binomial random variable: Let X be the number of
times a event A occurs in n independent trials. Let Ij
be the indicator function for event A in the jth trial.
Then
X = I1 + I2 + · · · + In
Graduate Institute of Communication Engineering, National Taipei University
21
Y. S. Han
Random Variables
and
P [X = k] =
where p = P [Ij = 1].
n
k
pk (1 − p)n−k ,
Graduate Institute of Communication Engineering, National Taipei University
22
Y. S. Han
Random Variables
23
• Geometric random variable: Count the number M
of independent Bernoulli trials until the first success of
event A.
P [M = k] = (1 − p)k−1 p
k = 1, 2, . . . ,
where p = P [A].
Graduate Institute of Communication Engineering, National Taipei University
Y. S. Han
Random Variables
24
• Uniform random variable
• Exponential random variable
(
0
x<0
fX (x) =
λe−λx x ≥ 0
FX (x) =
(
0
x<0
1 − e−λx x ≥ 0
λ: rate at which events occur.
Graduate Institute of Communication Engineering, National Taipei University
Y. S. Han
Random Variables
Graduate Institute of Communication Engineering, National Taipei University
25
Y. S. Han
Random Variables
26
• Gaussian (Normal) random variable:
The pdf is
1
−(x−m)2 /2σ 2
√
e
fX (x) =
2πσ
− ∞ < x < ∞,
where m and σ are real numbers.
The cdf is
Z x
1
−(x′ −m)2 /2σ 2
√
dx′ .
P [X ≤ x] =
e
2πσ −∞
Change variable t = (x′ − m)/σ and we have
Z (x−m)/σ
x−m
1
−t2 /2
e
dt = Φ
,
FX (x) = √
σ
2π −∞
Graduate Institute of Communication Engineering, National Taipei University
Y. S. Han
Random Variables
where
1
Φ(x) = √
2π
Z
x
−t2 /2
e
27
dt
−∞
Graduate Institute of Communication Engineering, National Taipei University
Y. S. Han
Random Variables
Graduate Institute of Communication Engineering, National Taipei University
28
Y. S. Han
Random Variables
29
Q-function is defined by
1
Q(x) = 1 − Φ(x) = √
2π
Z
∞
−t2 /2
e
dt.
x
Q-function is the probability of “tail” of the pdf.
Q(0) = 1/2
and
Q(−x) = 1 − Q(x).
Q(x) can be obtained by look-up tables.
Graduate Institute of Communication Engineering, National Taipei University
Y. S. Han
Random Variables
Example: A communication system accepts a positive
voltage V as input and output a voltage Y = αV + N ,
where α = 10−2 and N is a Gaussian random variable with
parameters m = 0 and σ = 2. Find the value of V that
gives P [Y < 0] = 10−6 .
Sol:
P [Y < 0] = P [αV + N < 0] = P [N < −αV ]
αV
−αV
=Q
= 10−6 .
= Φ
σ
σ
From the Q-function table, we have αV /σ = 4.753. Thus,
V = (4.753)σ/α = 950.6.
Graduate Institute of Communication Engineering, National Taipei University
30
Y. S. Han
Random Variables
3.5 Functions of a Random Variable
• Let X be a random variable. Define another random
variable Y = g(X). Example: Let the function
h(x) = (x)+ be defined as
(
0 x<0
+
(x) =
.
x x≥0
• Let B = {x : g(x) ∈ C}. The probability of event C is
P [Y ∈ C] = P [g(X) ∈ C] = P [X ∈ B].
• Three types of equivalent events are useful in
determining the cdf and pdf:
Graduate Institute of Communication Engineering, National Taipei University
31
Y. S. Han
Random Variables
1. Discontinuity case: {g(X) = yk };
2. cdf: {g(X) ≤ y};
3. pdf: {y < g(X) ≤ y + h}.
Graduate Institute of Communication Engineering, National Taipei University
32
Y. S. Han
Random Variables
Example: Let X be a sample voltage of a speech waveform, and
suppose that X has a uniform distribution in the interval [−4d, 4d].
Let Y = q(X), where the quantizer input-output characteristic is
shown below. Find the pmf for Y .
Graduate Institute of Communication Engineering, National Taipei University
33
Y. S. Han
Random Variables
Sol: The event {Y = q} for q in SY is equivalent to the event
{X ∈ Iq }, where Iq is an interval of points mapped into the
Graduate Institute of Communication Engineering, National Taipei University
34
Y. S. Han
Random Variables
representation point p. The pmf of Y
Z
fX (t)dt = 1/8
P [Y = q] =
35
for all q.
Iq
Graduate Institute of Communication Engineering, National Taipei University
Y. S. Han
Random Variables
Example: Let the random variable Y be defined by
Y = aX + b,
where a is a nonzero constant. Suppose that X has cdf FX (x), find
FY (y).
Sol: {Y ≤ y} and A = {aX + b ≤ y} are equivalent event. If a > 0
then A = {X ≤ (y − b)/a}, and thus
y−b
y−b
= FX
a > 0.
FY (y) = P X ≤
a
a
If a < 0, then A = {X ≥ (y − b)/a} and
y−b
y−b
FY (y) = P X ≥
= 1 − FX
.
a
a
Graduate Institute of Communication Engineering, National Taipei University
36
Y. S. Han
Random Variables
Therefore, we have


fY (y) =

y−b
a y−b
1
f
X
−a
a
1
a fX
a>0
1
=
fX
|a|
a<0
37
y−b
a
.
Graduate Institute of Communication Engineering, National Taipei University
Y. S. Han
Random Variables
Graduate Institute of Communication Engineering, National Taipei University
38
Y. S. Han
Random Variables
Example: Let X be a Gaussian random variable with mean m and
standard deviation σ:
1
−(x−m)2 /2σ 2
e
fX (x) = √
2πσ
−∞<x<∞
Let Y = aX + b. Find the pdf of Y .
Sol: From previous example, we have
1
−(y−b−am)2 /2(aσ)2
.
e
fY (y) = √
2π|a|σ
Y also has a Gaussian distribution with mean am + b and standard
deviation |a|σ.
Graduate Institute of Communication Engineering, National Taipei University
39
Y. S. Han
Random Variables
40
Example: Let random variable Y be defined by
Y = X 2,
where X is a continuous random variable. Find the cdf and pdf of Y .
Sol: The event {Y ≤ y} occurs when {X 2 ≤ y} or equivalently
√
√
{− y ≤ X ≤ y} for y nonnegative. The event is null when y is
negative. Then

 0
y<0
FY (y) =
 FX (√y) − FX (−√y) y ≥ 0
and
fY (y)
=
=
√
√
fX ( y) fX (− y)
−
√
√
2 y
−2 y
√
√
fX ( y) fX (− y)
+
.
√
√
2 y
2 y
y>0
Graduate Institute of Communication Engineering, National Taipei University
Y. S. Han
Random Variables
Graduate Institute of Communication Engineering, National Taipei University
41
Y. S. Han
Random Variables
• Consider Y = g(X) as shown below
• Consider the event Cy = {y < Y < y + dy}. Let Bx be its
equivalence in the x-axis.
• As shown in the figure, g(x) = y has three solutions and
Bx
= {x1 < X < x1 + dx1 } ∪ {x2 < X < x2 + dx2 }
Graduate Institute of Communication Engineering, National Taipei University
42
Y. S. Han
Random Variables
∪{x3 < X < x3 + dx3 }.
Thus,
P [Cy ] = fY (y)|dy| = P [Bx ] = fX (x1 )|dx1 |+fX (x2 )|dx2 |+fX (x3 )|dx3 |.
In general, we have
X fX (x) X
dx fY (y) =
=
fX (x) .
|dy/dx| x=xk
dy x=xk
k
k
Graduate Institute of Communication Engineering, National Taipei University
43
Y. S. Han
Random Variables
Example : Let Y = X 2 . For Y ≥ 0, the equation y = x2 has two
√
√
solutions, x0 = y and x1 = − y. Since dy/dx = 2x, we have
√
√
fX ( y) fX (− y)
+
.
fY (y) =
√
√
2 y
2 y
Example: Let Y = cos(X), where X is uniformly distributed in the
interval (0, 2π]. Find the pdf of Y .
Graduate Institute of Communication Engineering, National Taipei University
44
Y. S. Han
Random Variables
Sol: Two solutions in the interval, x0 = cos−1 (y) and x1 = 2π − x0 .
p
dy −1
= − sin(x0 ) = − sin(cos (y)) = − 1 − y 2 .
dx x0
Graduate Institute of Communication Engineering, National Taipei University
45
Y. S. Han
Random Variables
Since fX (x) = 1/(2π),
fY (y) =
=
1
2π
π
The cdf of Y is
FY (y) =



 0
1
2



1
p
p
1 − y2
1
1−
+
y2
sin−1 y
π
+
1
2π
p
1 − y2
for − 1 < y < 1.
y < −1
−1 ≤ y ≤ 1
y > 1.
Graduate Institute of Communication Engineering, National Taipei University
46
Y. S. Han
Random Variables
47
6
5
3
Y
f (y)
4
2
1
0
−1
−0.8
−0.6
−0.4
−0.2
0
y
0.2
0.4
0.6
0.8
1
Graduate Institute of Communication Engineering, National Taipei University
Y. S. Han
Random Variables
3.6 Expected Value of Random Variables
Graduate Institute of Communication Engineering, National Taipei University
48
Y. S. Han
Random Variables
The Expected Value of X
• The expected value or mean of a random
variable X is defined by
Z ∞
E[X] =
tfX (t)dt
−∞
• If X is a discrete random variable, then
X
xk pX (xk )
E[X] =
k
• Note that E[X] may not converge.
Graduate Institute of Communication Engineering, National Taipei University
49
Y. S. Han
Random Variables
50
• The mean for a uniform random variable between a
and b is given by
Z b
a+b
t
dt =
E[X] =
2
a b−a
E[X] is the midpoint of the interval [a, b].
• If the pdf of X is symmetric about a point m, then
E[X] = m. That is, when
fX (m − x) = fX (m + x),
we have
0=
Z
+∞
−∞
(m − t)fX (t)dt = m −
Z
+∞
tfX (t)dt.
−∞
Graduate Institute of Communication Engineering, National Taipei University
Y. S. Han
Random Variables
• The pdf of a Gaussian random variable is symmetric at
x = m. Therefore, E[X] = m.
Graduate Institute of Communication Engineering, National Taipei University
51
Y. S. Han
Random Variables
Exercise:
Show that if X is a nonnegative random variable, then
Z ∞
(1−FX (t))dt if X continuous and nonnegative
E[X] =
0
and
E[X] =
∞
X
P [X > k]
if X nonnegative, integer-valued.
k=0
Graduate Institute of Communication Engineering, National Taipei University
52
Y. S. Han
Random Variables
Expected value of Y = g(X)
• Let Y = g(X), where X is a random variable with pdf
fX (x).
• Y is also a random variable.
• Mean of Y is
E[Y ] =
Z
+∞
g(x)fX (x)dx.
−∞
Graduate Institute of Communication Engineering, National Taipei University
53
Y. S. Han
Random Variables
Variance of X
• Variance of the random variable X is defined by
VAR[X] = E[(X − E[X])2 ].
• Standard deviation of X
STD[X] = VAR[X]1/2
−−
measure of the spread of
a distribution.
• Simplification
VAR[X] = E[X 2 − 2E[X]X + E[X]2 ]
= E[X 2 ] − 2E[X]E[X] + E[X]2
= E[X 2 ] − E[X]2
Graduate Institute of Communication Engineering, National Taipei University
54
Y. S. Han
Random Variables
55
Example: Find the variance of the random variable X
that is uniformly distributed in the interval [a, b].
E[X] = (a + b)/2,
and
1
VAR[X] =
b−a
Z b
a
a+b
x−
2
2
dx
Let y = (x − (a + b)/2). Then
Z (b−a)/2
2
1
(b
−
a)
VAR[X] =
y 2 dy =
.
b − a −(b−a)/2
12
Graduate Institute of Communication Engineering, National Taipei University
Y. S. Han
Random Variables
56
Example: Find the variance of a Gaussian random
variable.
√
Multiply the integral of the pdf of X by 2πσ to obtain
Z +∞
√
−(x−m)2 /2σ 2
dx = 2πσ.
e
−∞
Differentiate both sides with respect to σ to get
Z +∞ 2
√
(x − m)
−(x−m)2 /2σ 2
dx = 2π.
e
3
σ
−∞
Then
1
VAR[X] = √
2πσ
Z
+∞
−∞
2 −(x−m)2 /2σ 2
(x − m) e
dx = σ 2 .
Graduate Institute of Communication Engineering, National Taipei University
Y. S. Han
Random Variables
• Properties
Let c be a constant. Then
VAR[c] = 0,
VAR[X + c] = VAR[X],
VAR[cX] = c2 VAR[X].
• nth moment of the random variable X is given by
Z +∞
xn fX (x)dx.
E[X n ] =
−∞
Graduate Institute of Communication Engineering, National Taipei University
57
Y. S. Han
Random Variables
58
3.7 Markov and Chebyshev Inequalities
Markov Inequality
• Suppose X is a nonnegative random variable with
mean E[X]. Then
E[X]
P [X ≥ a] ≤
a
for X nonnegative
Since
E[X] =
≥
Z
a
tfX (t)dt +
Z0 ∞
a
Z
a
∞
tfX (t)dt ≥
Z
∞
tfX (t)dt
a
afX (t)dt = aP [X ≥ a].
Graduate Institute of Communication Engineering, National Taipei University
Y. S. Han
Random Variables
Chebyshev Inequality
• Consider random variable X with E[X] = m and VAR[X] = σ 2 .
Then
σ2
P [|X − m| ≥ a] ≤ 2 .
a
• Proof: Let D2 = (X − m)2 . Markov inequality for D2 gives
E[(X − m)2 ]
σ2
P [D ≥ a ] ≤
= 2.
2
a
a
2
2
• {D2 ≥ a2 } and {|X − m| ≥ a} are equivalent events.
Graduate Institute of Communication Engineering, National Taipei University
59