Download Lecture Notes 2 Random Variables • Definition • Discrete Random

Lecture Notes 2
Random Variables
EE278
Prof. B. Prabhakar
Statistical Signal Processing
Autumn 02-03
• Definition
• Discrete Random Variables
• Continuous Random Variables
• Functions of a Random Variable
c
Copyright °2000–2002
Abbas El Gamal
2-1
EE 278: Random Variable
Random Variable
• A random variable (r.v.) X is a real-valued function X(ω) over the
sample space Ω, i.e., X : Ω → R
Ω
ω
X(ω)
• Notations:
– Always use upper case letters for random variables (X, Y , . . .)
– Always use lower case letters for values of random variables:
X = x means that the random variable X takes on the value x
EE 278: Random Variable
2-2
• Examples:
1. n coin flips: Here Ω = {H, T }n, define the random variable
X ∈ {0, 1, 2, . . . , n}
to be the number of heads
2. Let Ω = R, define the two random variables
a. X = ω,
(
b. Y =
1 for ω ≥ 0
−1 otherwise
3. Packet arrival times in the interval (0, T ]: Here Ω is the set of all
finite length strings (t1, t2, . . . , tn) ∈ (0, T ]∗, define the random
variable X to be the length of the string n = 0, 1, . . .
2-3
EE 278: Random Variable
Specifying a Random Variable
• Specifying a random variable means being able to determine the
probability that X ∈ A for any set A ⊂ R, e.g., any interval
• To do so, we consider the inverse image of the set A under X(ω),
{w : X(ω) ∈ A}
R
set A
inverse image of A under X(ω)
• So, X ∈ A iff ω ∈ {w : X(ω) ∈ A}, thus
P({X ∈ A}) = P({w : X(ω) ∈ A}), or in short
P{X ∈ A} = P{w : X(ω) ∈ A}
EE 278: Random Variable
2-4
Discrete Random Variables
• A random variable is said to be discrete if for some countable set
X ⊂ R, i.e., X = {x1, x2, . . .}, P{X ∈ X } = 1
• Examples 1, 2-b, and 3, are discrete random variables
x1x2 x3 . . . xn . . .
R
Ω
• Here X(ω) partitions Ω into the sets {ω : X(ω) = xi}, for i = 1, 2, . . ., so
to specify X, it suffices to know P{X = xi} for all i
• A discrete random variable is thus completely specified by its probability
mass function (pmf)
pX (x) = P{X = x}, for all x ∈ X
2-5
EE 278: Random Variable
• Clearly pX (x) ≥ 0 and
P
x∈X
pX (x) = 1
• So pX (x) can be simply viewed as a probability measure over a discrete
sample space (even though the original sample space may be continuous
as in examples 2-b and 3)
• The probability of any set A ⊂ R is given by
X
pX (x)
P{X ∈ A} =
x∈A∩X
• Notation: We use X ∼ pX (x) or simply X ∼ p(x) to mean that the
discrete random variable X has pmf pX (x) or p(x)
EE 278: Random Variable
2-6
Famous Discrete Random Variables
• Bernoulli r.v.: X ∼ Br(p), for 0 ≤ p ≤ 1 has pmf
pX (1) = p, and pX (0) = 1 − p
• Geometric r.v.: X ∼ Geom(p), for 0 ≤ p ≤ 1 has pmf
pX (k) = p(1 − p)k−1, for k = 1, 2, . . .
This r.v. represents, for example, the number of coin flips until the first
heads shows up (assuming independent coin flips)
• Binomial r.v.: X ∼ B(n, p), for integer n > 0, and 0 ≤ p ≤ 1 has pmf
µ ¶
n k
pX (k) =
p (1 − p)(n−k), for k = 0, 1, 2, . . . , n
k
This r.v. represents, for example, the number of heads in n independent
coin flips
• Poisson r.v.: X ∼ Poisson(λ), for λ > 0 has pmf
pX (k) =
λk −λ
e , for k = 0, 1, 2, . . .
k!
2-7
EE 278: Random Variable
This represents the number of random events in interval (0, 1], e.g.,
arrivals of packets, photons, customers, etc
f
l
Continuous Random Variables
• To specify a random variable, we need to be able to determine
P{X ∈ A} for any set A ⊂ R, i.e., any set generated by countable
unions, intersections, and complements of intervals
• Thus to specify X, it suffices to specify P{X ∈ (a, b]} for all intervals,
the probability of any other set can then be determined using the
axioms of probability
• Equivalently, to specify a random variable it suffices to specify its
cumulative distribution function (cdf)
FX (x) = P{X ≤ x}, for all x ∈ R
FX (x)
1
x
2-9
EE 278: Random Variable
• Properties of FX (x):
1. FX (x) ≥ 0 is monotonically nondecreasing, i.e., if a > b then
FX (a) ≥ FX (b)
2. limx→∞ FX (x) = 1 and limx→−∞ FX (x) = 0
3. FX (x) is right continuous, i.e., limx→a+ FX (x) = FX (a)
4. P{X = a} = FX (a) − FX (a−)
5. P{X ∈ A} for any Borel set A can be determined from FX (x)
• For a discrete random variable FX (x) consists of a countable set of steps
• A random variable is said to be continuous if its cdf FX (x) is a
continuous function
• Examples:
F1
1
x
EE 278: Random Variable
2-10
F2
1
x
F3
1
x
F4
1
x
• If FX (x) is continuous and differentiable (except over a countable set),
then X has a probability density function (pdf) fX (x) such that
Z x
fX (α)dα
FX (x) =
−∞
2-11
EE 278: Random Variable
• If FX (x) is differentiable everywhere, then
fX (x) =
P{x < X ≤ x + ∆x}
dFX (x)
= lim
∆x→0
dx
∆x
• Properties of fX (x):
1. fX (x) ≥ 0
R∞
2. −∞ fX (x)dx = 1
3. For any event A ∈ R
Z
P{X ∈ A} =
for example P{x1 < X ≤ x2} =
fX (x)dx,
x∈A
R x2
x1 fX (x)dx
• Important note: fX (x) should not be interpreted as the probabilty that
X = x, in fact it is not a probability measure, e.g., it can be > 1
• Notation: X ∼ fX (x) (or f (x)) means that X has pdf fX (x)
• Remark: Using δ(.) functions, we can define pdf for a r.v. with
discontinuous cdf, e.g., a discrete r.v.
EE 278: Random Variable
2-12
Famous Continuous Random Variables
• Uniform r.v.: X ∼ U[a, b], for b > a has pdf
(
1
for a ≤ x ≤ b
b−a
f (x) =
0
otherwise
• Exponential r.v.: X ∼ Exp(λ), for λ > 0 has pdf
(
λe−λx for x ≥ 0
f (x) =
0
otherwise
This r.v. represents interarrival time in a queue, e.g., time between two
consecutive packet or customer arrivals, also service time in a queue, and
lifetime of a particle, etc
• Example: Let X ∼ Exp(0.1) be the service time of customers at a bank
(in minutes). The person ahead of you has been served for 10 minutes.
What is the probability that you will wait another 10 minutes or more
before getting served?
2-13
EE 278: Random Variable
We want to find P{X > 20|X > 10}
By definition
P{X > 20, X > 10}
P{X > 10}
P{X > 20}
=
P{X > 10}
e−2
= −1
e
= e−1,
P{X > 20|X > 10} =
but P{X > 10} = e−1, i.e., the conditional probability of waiting more
than 10 miutes is the same as the unconditional probability of waiting
more than 10 minutes !
• This is because the exponential r.v. is memoryless, which in general
means that, for any 0 ≤ x1 < x2
P{X > x2|X > x1} = P{X > x2 − x1}
EE 278: Random Variable
2-14
Gaussian Random Variable
• Gaussian r.v.: X ∼ N (µ, σ 2) has pdf
fX (x) = √
1
−
(x−µ)2
2σ 2
e
,
2πσ 2
where µ is the mean and σ is the standard deviation
N (µ, σ 2)
x
µ
• Gaussian r.v.s are frequently encountered in nature, e.g., thermal and
shot noise in electronic devices are gaussian, and very frequently used in
modelling various social, biological, and other phenomena (a lot more on
gaussian r.v.s later)
2-15
EE 278: Random Variable
• The Φ, Q, and erfc functions:
Let X ∼ N (0, 1), and define its cdf as
Z x
1 − ξ2
√
e 2 dξ
Φ(x) =
2π
−∞
Also define the function
Q(x) = 1 − Φ(x)
N (0, 1)
Q(x)
x
As we shall soon see, the Q(·) function can be used to quickly compute
P{X > a} for any gaussian r.v. X
√
The function erfc(x) = 2Q( 2x), for x > 0
EE 278: Random Variable
2-16
Functions of a Random Variable
• We are often given a r.v. with known distribution (pmf, cdf, or pdf), a
function y = g(x), and would like to specify the random variable
Y = g(X)
• If X ∼ pX (x), i.e., discrete r.v., then Y is also discrete with pmf
X
pY (y) =
pX (x)
{x:g(x)=y}
g(xi) = y
x1 x2
x3
...
y1 y2 . . . y . . .
• If X ∼ fX (x), i.e., a continuous r.v., and the function g is differentiable,
then we can find the pdf of Y as follows
2-17
EE 278: Random Variable
Note that
P{y < Y ≤ y + ∆y}
∆y→0
∆y
So we first find the inverse image of the set (y, y + ∆y]
fY (y) = lim
x
y
y + ∆y
y
{x : y < g(x) ≤ y + ∆y}
Thus
P{y < Y ≤ y + ∆y} = P{x : y < g(x) ≤ y + ∆y},
or
fY (y)∆y ≈ P{x : y < g(x) ≤ y + ∆y}
from which we can find fY (y) ( as we shall demonstrate in the following
examples)
EE 278: Random Variable
2-18
• Example: Linear Function of a r.v. Let Y = aX + b
y
y + ∆y
y
x
y−b y+∆y−b
a
a
fY (y)∆y ≈ P{x
½ : y < g(x) ≤ y + ∆y} ¾
y − b ∆y
y−b
<X≤
+
= P
a
a
a
¶
µ
y − b ∆y
≈ fX
a
|a|
Thus as we let ∆y → 0 we get
¶
µ
1
y−b
fY (y) = fX
|a|
a
2-19
EE 278: Random Variable
• Example: Quadratic Function of a r.v. Let Y = X 2
y
y + ∆y
y
√
√
y
x
√
y + 2∆y
y
fY (y)∆y ≈ P{x
¾
½ : y < g(x) ≤ y + ∆y}
∆y
∆y
√
√
√
√
y < X ≤ y + √ or − y − √ < X ≤ − y
= P
2 y
2 y
µ
¶
1
1
√
√
≈
√ fX ( y) + √ fX (− y) ∆y
2 y
2 y
Thus
EE 278: Random Variable
1
√
√
fY (y) = √ (fX ( y) + fX (− y))
2 y
2-20
• In general, let X ∼ fX (x) and Y = g(X) be differentiable, then
fY (y) =
k
X
fX (xi)
i=1
|g 0(xi)|
,
where x1, x2, . . . are the solutions of the equation y = g(x) (in terms of
y) and g 0(xi) is the derivative of g evaluated at xi
• If the cdf of X is given and we wish to find the cdf of Y
y
x
y
{x : g(x) ≤ y}
then we use
FY (y) = P{Y ≤ y} = P{x : g(x) ≤ y}
2-21
EE 278: Random Variable
• Example: Let X ∼ F (x) (cdf), where F (x) is continuous, and define
Y = F (X). Find the cdf of Y .
To do so, consider
FY (y) =
=
=
=
P{Y ≤ y}
P{F (X) ≤ y}
©
ª
P X ≤ F −1(y)
¢
¡
F F −1(y) = y, for 0 ≤ y ≤ 1
Thus Y ∼ U [0, 1] !!
• Note: Even though we assumed here that F (x) is invertible, it does not
need to be — if F (x) = a, a constant over some interval, i.e., the
probability that X lies in this interval is zero, then without loss of
generality we can take F −1(a) to be the leftmost point of the interval
EE 278: Random Variable
2-22