Download Solutions5

6.434J/16.391J
Statistics for Engineers and Scientists
MIT, Spring 2006
Apr 11
Handout #13
Solution 5
Problem 1: Let a > 0 be a known constant, and let θ > 0 be a parameter.
Suppose X1 , X2 , . . . , Xn is a sample from a population with one of the following
densities.
(a) The beta, β(θ, 1), density: fX (x | θ) = θxθ−1 ,
for 0 < x < 1.
a
(b) The Weilbull density: fX (x | θ) = θaxa−1 e−θx ,
(c) The Pareto density: fX (x | θ) =
θaθ
,
x(θ+1)
for x > 0.
for x > a.
In each case, find a real-valued sufficient statistic for θ.
Solution Let X (X1 , X2 , . . . , Xn ) be a collection of i.i.d. random variables
Xi ’s, and let x (x1 , x2 , . . . , xn ) be a collection of observed data.
(a) For any x, the joint pdf is

θn (x x · · · x )θ−1 ,
1 2
n
fX (x | θ) =
0,
if ∀i, 0 < xi < 1
otherwise;
= θn (x1 x2 · · · xn )θ−1 × I(0,1) (x1 )I(0,1) (x2 ) · · · I(0,1) (xn ) .
g(T (x) | θ)
h(x)
Factorization theorem implies that
T (x) x1 x2 · · · xn
is a sufficient statistic for θ.
(b) For any x, the joint pdf is

θn an (x x · · · x )a−1 e−θ
1 2
n
fX (x | θ) =
0,
−θ
= θ n e n
i=1
n
i=1
xa
i
,
if ∀i, xi > 0;
otherwise;
xa
i
g(T (x) | θ)
× an (x1 x2 · · · xn )a−1
×I(0,∞) (x1 )I(0,∞) (x2 ) · · · I(0,∞) (xn ) .
h(x)
1
Factorization theorem implies that
T (x) n
xai
i=1
is a sufficient statistic for θ.
(c) For any x, the joint pdf is

θ n anθ

, if ∀i, xi > a;
(x1 x2 ···xn )θ+1
fX (x | θ) =
0,
otherwise;
=
θn anθ
× I(a,∞) (x1 )I(a,∞) (x2 ) · · · I(a,∞) (xn ) .
(x1 x2 · · · xn )θ+1 h(x)
g(T (x) | θ)
Factorization theorem implies that
T (x) x1 x2 · · · xn
is a sufficient statistic for θ.
Problem 2:
a) Let X1 , X2 , . . . , Xn be independent random variables, each uniformly distributed on the interval [−θ, θ], for some θ > 0. Find a sufficient statistic
for θ.
b) Let X1 , X2 , . . . , Xn be a random sample of size n from a normal N (θ, θ)
distribution, for some θ > 0. Find a sufficient statistic for θ.
Solution
a) For any x (x1 , x2 , . . . , xn ), the joint pdf is given by
 n
 1 , if ∀i, −θ ≤ xi ≤ θ;
2θ
fX (x | θ) =
0,
otherwise;
 n
 1 , if −θ ≤ min(x1 , . . . , xn ) and max(x1 , . . . , xn ) ≤ θ;
2θ
=
0,
otherwise;
=
1 n
I[−θ,∞) (min(x1 , . . . , xn ))I(−∞,θ] (max(x1 , . . . , xn )) × 1 .
2θ
g(T(x) | θ)
2
h(x)
Factorization theorem implies that
T(x) min(x1 , . . . , xn ), max(x1 , . . . , xn )
is jointly sufficient for θ.
b) For any x (x1 , x2 , . . . , xn ), the joint pdf is given by
1 n
n
2
1
e− 2θ i=1 (xi −θ)
fX (x) = √
2πθ
1 n
n
2
n
2
1
= √
e− 2θ ( i=1 xi −2θ i=1 xi +nθ )
2πθ
1 n
n
2
n
1
nθ
= √
e− 2θ i=1 xi + i=1 xi − 2
2πθ
1 n
1 n
n
n
2
1
nθ
= √
e i=1 xi × √
e− 2θ i=1 xi − 2 .
2π
θ
h(x)
g(T (x) | θ)
Factorization theorem implies that
T (x) n
x2i
i=1
is a sufficient statistic for θ.
Problem 3: Let X be the number of trials up to (and including) the first
success in a sequence of Bernoulli trials with probability of success θ, for 0 <
θ < 1. Then, X has a geometric distribution with the parameter θ:
Pθ {X = k} = (1 − θ)k−1 θ,
k = 1, 2, 3, . . . .
Show that the family of geometric distributions is a one-parameter exponential
family with T (x) = x.
[Hint: xα = eα ln x , for x > 0.]
Solution Recall that the pmf of a one-parameter (θ) exponential family is of
the form
p(x | θ) = h(x) eη(θ)T (x)−B(θ) ,
where x ∈ X . Rewriting the pmf of a Geometric random variable yields
Pθ {X = x} = e(x−1) ln(1−θ)+ln θ
= ex ln(1−θ)−(ln(1−θ)−ln θ) ,
3
where x ∈ {1, 2, 3, . . . }. Thus, the geometric distribution is a one-parameter
exponential family with
h(x) 1
η(θ) ln(1 − θ)
T (x) x
B(θ) ln(1 − θ) − ln θ
X {1, 2, 3, . . . }.
Problem 4: Let X1 , X2 , . . . , Xn be a random sample of size n from the truncated Bernoulli probability mass function (pmf),

p,
if x = 1;
P {X = x | p} =
(1 − p), if x = 0.
(a) Show that the joint pmf of X1 , X2 , . . . , Xn is a member of the exponential
family of distribution.
(b) Find a minimal sufficient statistic for p.
Solution
(a) Let x (X1 , X2 , . . . Xn ) denote the collection of i.i.d. Bernoulli random
variables. The joint pmf is given by
P {X = x | p} = px1 (1 − p)1−x1 px2 (1 − p)1−x2 · · · pxn (1 − p)1−xn
=p
=e
=e
n
i=1
xi
(ln p)
(1 − p)n−
n
i=1
xi
n
i=1
xi
e[ln(1−p)][n−
[ln p−ln(1−p)]
n
i=1
n
i=1
xi +n ln(1−p)
xi ]
,
for x ∈ {0, 1}n . Therefore, the joint pmf is a member of the exponential
family, with the mappings:
θ=p
h(x) = 1
η(p) = ln p − ln(1 − p)
T (x) =
n
xi
i=1
X = {0, 1}n .
B(p) = −n ln(1 − p)
(b) Let x, y ∈ {0, 1}n be given. Consider the likelihood ratio,
P {X = x | p}
= e[ln p−ln(1−p)][
P {X = y | p}
4
n
i=1
xi −
n
i=1
yi ]
.
Define a function k(x, y) h(x)/h(y) = 1, which is bounded and non-zero
for any x ∈ X and y ∈ X .
n
n
Note that x and y such that i=1 xi = i=1 yi are equivalent because
function k(x, y) satisfies the requirement of likelihood ratio partition.
n
Therefore, T (x) i=1 xi is a sufficient statistic.
Problem 5: Let X1 , X2 , . . . , Xm and Y1 , Y2 , . . . , Yn be two independent samples from N (µ, σ 2 ) and N (µ, τ 2 ) populations, respectively. Here, −∞ < µ < ∞,
σ 2 > 0, and τ 2 > 0. Find a minimal sufficient statistic for θ (µ, σ 2 , τ 2 ).
Solution Let X (X1 , X2 , . . . , Xm ) and Y (Y1 , Y2 , . . . , Yn ) denote the collections of random samples. The joint pdf (of Xj ’s and Yi ’s), evaluated at
x (x1 , x2 , . . . , xm ) and y (y1 , y2 , . . . , yn ), is given by
m (xj −µ)2 1 m
j=1
1 n − ni=1 (y2i −µ)2
2σ 2
2τ
· e−
· √
·e
fX,Y (x, y | θ) = √
2πσ 2
2πτ 2
1
= e− 2σ2
m
j=1
x2j − 2τ12
n
i=1
yi2 + σµ2
m
j=1
xj + τµ2
2
n
i=1
yi −B(µ,σ 2 ,τ 2 )
,
2
mµ
nµ
n
2
2
where B(µ, σ 2 , τ 2 ) m
2 ln 2πσ + 2 ln 2πτ + 2σ 2 + 2τ 2 .
Notice that the joint pdf belongs to the exponential family, so that the
minimal statistic for θ is given by
T(X, Y) m
Xj2 ,
j=1
n
i=1
Yi2 ,
m
j=1
Xj ,
n
Yi .
i=1
Note: One should not be surprised that the joint pdf belongs to the exponential family of distribution. Recall that Gaussian distribution is a member of the
exponential family of distribution and that random variables, Xi ’s and Yj ’s, are
mutually independent. Thus, their joint pdf belongs to the exponential family
as well.
Note: To derive the minimal sufficient statistic, one may alternatively consider
likelihood ratio partition.
The set D0 is defined to be
D0 (x, y) ∈ Rm+n for all µ, for all σ 2 > 0, for all τ 2 > 0
fX,Y x, y | µ, σ 2 , τ 2 = 0
=∅
(empty set).
5
Let (x, y) ∈
/ D0 and (v, w) ∈
/ D0 be given. Their likelihood ratio is given by
m
m
n
n
fX,Y (x, y | θ)
1 2 2 1 2 2 = exp − 2
xj −
vj − 2
yi −
wi
fX,Y (v, w | θ)
2σ j=1
2τ i=1
j=1
i=1
m
m
n
n
µ µ + 2
xj −
vj + 2
yi −
wi .
σ j=1
τ i=1
j=1
i=1
/ D0 are equivalent iff there exists a
By definition, (x, y) ∈
/ D0 and (v, w) ∈
function, 0 < k(·, ·, ·, ·) < ∞, which is independent of θ, such that
fX,Y (x, y | θ)
= k(x, y, v, w).
fX,Y (v, w | θ)
The likelihood ratio implies that (x, y) ∈
/ D0 and (v, w) ∈
/ D0 are equivalent if
and only if
m
x2j =
j=1
n
i=1
m
yi2 =
xj =
j=1
n
yi =
i=1
m
j=1
n
i=1
m
j=1
n
vj2 ,
(1)
wi2 ,
(2)
vj , and
(3)
wi ,
(4)
i=1
where function k(x, y, v, w) 1.
That is, (x, y) and (v, w) are in the same equivalent class iff conditions
(1)-(4) are satisfied. Then a representation of the equivalent class is given by
T(X, Y) m
j=1
Xj2 ,
n
Yi2 ,
i=1
m
j=1
Xj ,
n
Yi .
i=1
Thus, we have a minimal sufficient statistic, T(X, Y).
Problem 6: The two hypotheses about the probability density fX (x) of an
observed random variable X are
H1 :
H0 :
1 −|x|
e
,
2
1 2
1
fX (x) = √ e− 2 x ,
2π
fX (x) =
6
for any x
for any x.
(a) Find the likelihood ratio Λ(x).
(b) The test is of the form
H1
Λ(x) ≷ η .
H0
Compute the decision regions for various values of the threshold η.
Solution
(a) Let x ∈ R denote an observation. The likelihood ratio is given by
Λ (x) fX | H (x | H1 )
.
fX | H (x | H0 )
Substituting the densities of random variable X (under hypothesis H1 and
under hypothesis H0 ) yields the likelihood ratio
1 −|x|
2 e
Λ (x) = 1 2
√1 e− 2 x
2π
π 1 x2 −|x|
=
e2
.
2
(b) The decision region for hypothesis H1 , R1 , is the set of points x’s that
give rise to the output decision H1 :
R1 {x | the test decides H1 on input x}
= {x | Λ (x) > η}.
Similarly, the decision region for hypothesis H0 , R0 , is given by
R0 {x | the test decides H0 on input x}
= {x | Λ (x) ≤ η}
= R\R1 ,
where the symbol “\” denotes the set difference.
Substituting the expression of the likelihood ratio from part (a) yields the
following definition of decision region R1 :
π 1 x2 −|x|
e2
>η .
R1 = x 2
When 0 ≥ η, we will have R1 = R since ey > 0 ≥ η for any y. Thus, we
will consider the case when η > 0.
7
Taking natural log both sides of the inequality and writing x2 as |x|2 yields
1
η √2 2
R1 = x |x| − |x| − ln √
>0 .
2
π
√ When 1 + 2 ln η√π2 < 0, the decision region is empty (since the term,
b2 − 4ac, in the square root of the quadratic formula is negative).
√ π −1
2 , we will have a
When 1 + 2 ln η√π2 ≥ 0, or equivalently, η ≥
2 e
non-empty decision region,
√2 R1 = x |x| > 1 + 1 + 2 ln η
or
π
√2 |x| < 1 − 1 + 2 ln η
π
√
2
= x |x| > 1 + 1 + 2 ln η
π
=
x x > 1 +
(absolute value cannot be negative)
√2 or
1 + 2 ln η
π
√2 x < −1 − 1 + 2 ln η
.
π
Therefore, the decision region R1 is given by


R,




∅,
R1 =
√ √


(1
+
1 + 2 ln(η 2/ π), ∞)




(−∞, 1 − 1 + 2 ln(η √2/√π) ),
for η ≤ 0;
for 0 < η <
for
π
2
π
2
e−1/2 ;
e−1/2 ≤ η,
while the decision region R0 is given by
R0 = R\R1


∅,
for η ≤ 0;





R,
for 0 < η < π2 e−1/2 ;
= √ √

1
−
1 + 2 ln(η 2/ π),





 1 + 1 + 2 ln(η √2/√π) , for π e−1/2 ≤ η.
2
8