Download Self-normalized Limit Theorems in Probability and Statistics

Self-normalized Limit Theorems in Probability and Statistics
Qi-Man Shao
Hong Kong University of Science and Technology
and
University of Oregon
Abstract. The normalizing constants in classical limit theorems are usually sequences of real numbers.
Moment conditions or other related assumptions are necessary and sufficient for many classical limit theorems.
However, the situation becomes very different when the normalizing constants are sequences of random variables.
A self-normalized large deviation shows that no any moment condition is needed for such large deviation type
results. A self-normalized law of the iterated logarithm remains valid for all distributions in the domain of
attraction of a normal or stable law. This reveals that self-normalization preserves essential properties much
better than deterministic normalization does. This paper surveys briefly some recent developments on selfnormalized limit theorems.
1
1
Introduction
Let X, X1 , X2 , · · · be independent and identically distributed random variables. Put
Sn =
n
X
Xi and
i=1
Vn2
=
n
X
Xi2 .
(1)
i=1
It is well-known that moment conditions or other related conditions are necessary and sufficient for
many classical limit theorems. For example, the strong law of large numbers holds if and only if the
mean of X is finite; the central limit theorem holds if and only if EX 2 I(|X| ≤ x) is slowing varying as
x → ∞; and a necessary and sufficient condition for the large deviation is that the moment generating
function of X is finite in a neighborhood of zero. On the other hand, limit theorems for self-normalized
sums Sn /Vn put a totally new countenance upon the classical limit theorems. In contrast to the wellknown Hartman-Wintner law of the iterated logarithm and its converse by Strassen (1966), Griffin
and Kuelbs (1989) obtained a self-normalized law of the iterated logarithm for all distributions in the
domain of attraction of a normal or stable law. Shao (1997) showed that no moment conditions are
√
needed for a self-normalized large deviation result P (Sn /Vn ≥ x n) and that the tail probability of
Sn /Vn is Gaussian like when X1 is in the domain of attraction of the normal law and sub-Gaussian
like when X is in the domain of attraction of a stable law, while Giné, Götze and Mason (1997) proved
that the tails of Sn /Vn are uniformly sub-Gaussian when the sequence is stochastically bounded. Shao
(1999) established a Cramér type result for self-normalized sums only under a finite third moment
condition. These results strongly show that self-normalized partial sums preserve desirable properties
much better than non-randomized partial sums. Self-normalization is also commonly used in statistics.
Many statistical inferences require the use of classical limit theorems. However, these classical results
often involve some unknown parameters, one needs to first estimate the unknown parameters and then
substitute the estimators into the classical limit theorems. This commonly used practice is exactly
the self-normalization. A typical case is the Student t-statistic. The close relationship between the
Student t-statistic Tn and the self-normalized sum Sn /Vn can be seen below:
Tn :=
1/2
X̄
n−1
S √ = n
Vn n − (Sn /Vn )2
s/ n
and
{Tn ≥ t} =
1/2 Sn
n
≥t
,
Vn
n + t2 − 1
(2)
(3)
where X̄ is the sample mean and s is the sample standard deviation.
The main purpose of this paper is to give a brief survey on recent developments in the direction
of self-normalized limit theory. We will focus on the following topics
1. Self-normalized large deviations
2. Self-normalized saddlepoint approximations
3. Self-normalized moderate deviations
4. Cramér type large deviations for independent random variables
5. Self-normalized laws of the iterated logarithm
2
6. Limiting distributions of self-normalized sums
7. Weak invariance principle for self-normalized partial sum processes
8. Exponential inequalities for self-normalized processes
9. Applications to statistics
2
Self-normalized large deviations
Let X, X1 , X2 , · · · be a sequence of independent and identically distributed (iid) random variables.
The classical Chernoff large deviation [7] states that if
A) Eet0 X < ∞ for some t0 > 0,
then for every x > EX,
lim P
n→∞
1/n
Sn
≥x
= ρ(x),
n
(4)
where ρ(x) = inf t≥0 e−tx EetX .
Roughly speaking, this type of large deviation shows that the convergence rate in the law of large
numbers is exponential if the moment generating function is finite in a right neighbourhood of zero.
The latter is also necessary for an exponential scale. Essentially built on condition A), the area of
large deviations in finite dimensional spaces, as well as in abstract spaces, has been well developed,
and various applications in statistics (c.f., e.g., Bahadur (1971)), engineering, statistical mechanics and
applied probability have been found in recent years. We refer to de Acosta (1988), Stroock (1984),
Donsker and Varadhan (1987) and the book by Dembo and Zeitouni (1992), and references therein,
for more details.
On the other hand, the following result shows that a self-normalized large deviation remains valid
for arbitrary random variables without any moment conditions.
Theorem 1 [Shao (1997)] Assume that either E(X) ≥ 0 or E(X 2 ) = ∞. Then
lim P
n→∞
1/n
Sn
2
2 2
≥x
= sup inf e−tx Eet(2cX−c X )
1/2
Vn n
c≥0 t≥0
(5)
for x > E(X)/(E(X 2 ))1/2 , where E(X)/(E(X 2 ))1/2 = 0 if E(X 2 ) = ∞.
Remark 2.1 Note that for any random variable X, E(X 2 ) is either finite or infinite. When E(X 2 )
is finite, of course, E(|X|) is finite. It is reasonable to assume E(X) ≥ 0. In fact, when E(X) ≤ 0,
(5) remains true for x > 0.
A key observation of the proof of Theorem 1 is the following identity
∀ x ≥ 0, y ≥ 0, x1/2 y 1/2 = (1/2) inf (x/c + yc).
c>0
Thus, one can write
√
x nVn = (1/2) inf (Vn2 /c + cx2 n)
c>0
3
(6)
and
√
P (Sn /Vn ≥ x n) = P Sn ≥ (1/2) inf (x2 n/c + Vn2 c)
c>0
= P sup
c>0
n
X
(2cXi − c2 Xi2 ) ≥ x2 n .
(7)
i=1
Now for each fixed c > 0, {2cXi − c2 Xi2 , i ≥ 1} is a sequence of i.i.d. random variables with finite
moment generating function in a right neighborhood of zero. Applying the Chnorff large deviation
(4) yields
√
2
2 2
lim inf P (Sn /Vn ≥ x n)1/n ≥ sup inf e−tx E(et(2cX−c X ) ).
n→∞
c≥0 t≥0
As to the upper bound, intuitively, the order of the probability of the union of events should be close
to the maximum of the probability of the individual event. The detailed proof is much more involved.
Theorem 1 has been extended to high dimension by Dembo and Shao (1998a) and to self-normalized
empirical processes by Bercu, Gassiat and Rio (2002).
3
Self-normalized saddlepoint approximations
Let X, X1 , X2 , · · · be i.i.d random variables and denote X̄ the sample mean of {Xi , 1 ≤ i ≤ n}. Large
deviation result provides an exponential rate of convergence for tail probability. However, a more
fine-tuned approximation can be offered by saddlepoint approximations. Daniels (1954) showed that
the density function of X̄ satisfies
−n(τ x−κ(τ ))
fX̄ (x) = e
1/2
n
(1 + O(n−1 )),
2πκ00 (τ )
where τ is the saddlepoint satisfying κ0 (τ ) = x. Lugannani and Rice (1980) obtained the tail probability of X̄:
√
√
φ( nŵ) 1
1
P (X̄ ≥ x) = 1 − Φ( nŵ) + √
− + O(n−1 ) ,
û ŵ
n
where κ0 (τ ) = x, ŵ = {2[τ κ0 (τ ) − κ(τ )]}1/2 sign{τ }, û = τ [κ00 (τ )]1/2 , Φ and φ denote the standard
normal distribution function and density function, respectively. So, the error incurred by the saddlepoint approximation is O(n−1 ) as against the more usual O(n−1/2 ) associated with the normal
approximation. Another desirable feature of saddlepoint approximation is that the approximation is
quite satisfactory even when the sample size n is small. The book by Jensen (1995) gives a detailed
account of saddlepoint approximations and related techniques. However, a finite moment generating
function is an essential requirement for saddlepoint expansions. Daniels and Young (1991) derived
saddlepoint approximations for the tail probability of the Student t-statistic under the assumption
that the moment generating function of X 2 exists. Note that Theorem 1 holds without any moment
assumption. It is natural to ask whether the saddle point approximation is still valid without any
moment condition for the t statistic or equivalently, for the self-normalized sum Sn /Vn . Jing, Shao
2
and Zhou (2004) recently give an affirmative answer to this question. Let K(s, t) = ln EesX+tX ,
K11 (s, t) =
∂ 2 K(s, t)
∂ 2 K(s, t)
∂ 2 K(s, t)
,
K
(s,
t)
=
,
K
(s,
t)
=
.
12
22
∂s2
∂s∂t
∂t2
4
For 0 < x < 1, let t̂0 and a0 be solutions t and a to the equations
2
2
2
2)
EX 2 et(−2aX/x +X
EXet(−2aX/x +X )
=
a,
2
2
Eet(−2aX/x +X )
Eet(−2aX/x2 +X 2 )
=
a2
x2
It is proved in [35] that t̂0 < 0. Put ŝ0 = −2a0 t̂0 /x2 and define
Λ0 (x) = ŝ0 a0 + t̂0 a20 /x2 − K(ŝ0 , t̂0 ),
Λ1 (x) = 2t̂0 /x2 + (1, 2a0 /x2 )∆−1 (1, 2a0 /x2 )0 ,
where
∆=
K11 (ŝ0 , t̂0 )
K12 (ŝ0 , t̂0 )
K12 (ŝ0 , t̂0 )
K22 (ŝ0 , t̂0 )
.
2 [Jing, Shao and Zhou (2004)]. Assume EX = 0 or EX 2 = ∞ and that
RTheorem
∞ R∞
isX+itX 2 |r dsdt < ∞ for some r > 1. Then for 0 < x < 1
−∞ −∞ |Ee
√
√
√
φ( nw) 1
1
− + O(n−1 ) ,
P (Sn /Vn ≥ x n) = 1 − Φ( nw) − √
w v
n
where w =
4
p
1/2 Λ (x)1/2 .
2Λ0 (x), and v = (−t̂0 /2)1/2 x3/2 a−1
1
0 (det ∆)
Self-normalized moderate deviations
Let X, X1 , X2 , · · · be i.i.d random variables and let {xn , n ≥ 1} be a sequence of positive numbers
with xn → ∞ as n → ∞. It is known that
|Sn |
1
−2
lim xn ln P √ ≥ xn = −
n→∞
2
n
√
holds for any sequence {xn } with xn → ∞ and xn = o( n) if and only if EX = 0, EX 2 = 1 and
Eet0 |X| < ∞ for some t0 > 0. The sufficient part follows from the Cramér large deviation (c.f. Petrov
(1975) ). Following Shao (1989), the necessary part can be proved by studying the increments of
Sn . The next result shows again that the situation is quite different in the case of self-normalized
limit theorems. It tells us that the main term of the asymptotic probability of P (Sn ≥ xn Vn,2 ) is
√
distribution free as long as X is in the domain of attraction of a normal law and xn = o( n).
Theorem 3 [Shao (1997)] Let {xn , n ≥ 1} be a sequence of positive numbers with xn → ∞ and
√
xn = o( n) as n → ∞. If EX = 0 and EX 2 I{|X| ≤ x} is slowly varying as x → ∞, then
Sn
1
−2
lim x ln P
≥ xn = − .
n→∞ n
Vn,2
2
Similar results to that of Theorem 3 remain valid when X is in the domain of attraction of a stable
law.
5
Theorem 4 [Shao (1997)] Assume that there exist 0 < α < 2, c1 ≥ 0, c2 ≥ 0, c1 + c2 > 0 and a
slowly varying function h(x) such that
c2 + o(1)
c1 + o(1)
h(x) and P (X ≤ −x) =
h(x) as x → ∞.
xα
xα
Moreover, assume that EX = 0 if 1 < α < 2, X is symmetric if α = 1 and that c1 > 0 if 0 < α < 1.
Let {xn , n ≥ 1} be a sequence of positive numbers with xn → ∞ and xn = o(n1/2 ) as n → ∞. Then,
we have
Sn
−2
lim x ln P
≥ xn = −β(α, c1 , c2 ),
n→∞ n
Vn
P (X ≥ x) =
where β(α, c1 , c2 ) is the solution of Γ(β, α, c1 , c2 ) = 0 and

Z ∞
Z ∞
2
2

1 − 2x − e−2x−x /β
1 + 2x − e2x−x /β


dx
+
c
dx if 1 < α < 2,
c

2
1

xα+1
xα+1

0
0



Z ∞
2
2

2 − e2x−x /β − e−2x−x /β
Γ(β, α, c1 , c2 ) =
c1
dx
if α = 1,

x2

0



Z ∞
Z ∞
2
2


1 − e2x−x /β
1 − e−2x−x /β


dx
+
c
dx
if 0 < α < 1.
 c1
2
xα+1
xα+1
0
0
In particular, if X is symmetric, then
lim x−2
n→∞ n
∞
Z
where β(α) is the solution of
0
5
ln P
2 /β
2 − e2x−x
Sn
≥ xn
Vn
− e−2x−x
xα+1
= −β(α),
2 /β
dx = 0.
Cramér type large deviations for independent random variables
Let X1 , X2 , · · · be a sequence of independent random variables with EXi = 0 and 0 < EXi2 < ∞ for
i ≥ 1. Set
n
n
n
X
X
X
Sn =
Xi ,
Bn2 =
EXi2 ,
Vn2 =
Xi2 .
i=1
i=1
i=1
It is well-known that the central limit theorem holds if the Lindeberg condition is satisfied. There
are mainly two approaches for estimating the error of the normal approximation. One approach
is to study the absolute error in the central limit theorem via Berry-Esseen bounds or Edgeworth
expansions. Another approach is to estimate the relative error of P (Sn ≥ xBn ) to 1 − Φ(x). One of
the typical results in this direction is the so-called Cramér large deviation. If X1 , X2 , · · · are a sequence
of i.i.d. random variables with zero means and a finite moment-generating function EetX1 < ∞ for t
in a neighborhood of zero, then for x ≥ 0 and x = o(n1/2 )
P (Sn ≥ xBn )
x
1+x
2
= exp x λ( √ ) 1 + O( √ )
1 − Φ(x)
n
n
where λ(t) is the so-called Cramér’s series (see [Petrov (1975), Chapter VIII] for details). In particular,
1/2
if Eet|X1 | < ∞ for some t > 0, then
P (Sn ≥ xBn )
→1
1 − Φ(x)
6
(8)
1/2
holds uniformly for x ∈ (0, o(n1/6 )). Note that the moment condition Eet|X1 |
< ∞ is necessary.
Similar results are also available for independent but not necessarily identically distributed random
variables under a finite moment-generating function condition.
Shao (1999) established a (8) type result for self-normalized sums only under a finite third-moment
condition. More precisely, he showed that if E|X1 |2+δ < ∞ for 0 < δ ≤ 1, then
P (Sn ≥ xVn )
→1
1 − Φ(x)
(9)
holds uniformly for x ∈ (0, o(nδ/(2(2+δ)) )). Recently, several papers have focused on the self-normalized
limit theorems for independent but not necessarily identically distributed random variables. Bentkus,
Bloznelis and Götze (1996) obtained the following Berry-Esseen bound:
|P (Sn /Vn ≥ x) − (1 − Φ(x))|
n
n
X
X
−2
2
−3
≤ A Bn
EXi I{|Xi |>Bn } + Bn
E|Xi |3 I{|Xi |≤Bn } ,
i=1
i=1
where A is an absolute constant. Assuming only finite third moments, Wang and Jing (1999) derived
exponential non-uniform Berry-Esseen bounds. Chistyakov and Götze (2003) refined Wang and Jing’
results and obtained the following result among others: If X1 , X2 , · · · are symmetric independent
random variables with finite third moments, then
n
X
3 −3
P (Sn /Vn ≥ x) = (1 − Φ(x)) 1 + O(1)(1 + x) Bn
E|Xi |3
(10)
i=1
P
for 0 ≤ x ≤ Bn /( ni=1 E|Xi |3 )1/3 , where O(1) is bounded by an absolute constant.
Result (10) is useful because it provides not only the relative error but also a Berry-Esseen rate of
convergence. It should be noted that if Xi is symmetric, then
n
X
Sn /Vn and (
εi Xi )/Vn
i=1
have the same distribution, where {εi , i ≥ 1} is a Rademacher sequence that is independent of {Xi , i ≥
1}. Hence given {Xi , 1 ≤ i ≤ n}, the problem reduces to estimate the tail probability of the partial
sum of independent random variables εi Xi /Vn , 1 ≤ i ≤ n. So, the assumption of symmetry not only
takes away the main difficulty in proving a self-normalized limit theorem but also limits its potential
applications. Jing, Shao and Wang (2003) recently obtained a Cramér-type large deviation for general
independent random variables. In particular, they show that (10) remains valid for non-symmetric
independent random variables. Let
∆n,x =
n
n
(1 + x)3 X
(1 + x)2 X
2
EX
I{|X
|
>
B
/(1
+
x)}
+
E|Xi |3 I{|Xi | ≤ Bn /(1 + x)}
i
n
i
Bn2
Bn3
i=1
i=1
for x ≥ 0.
Theorem 5 [Jing, Shao and Wang (2003)]. There is an absolute constant A (> 1) such that
P (Sn ≥ xVn )
= eO(1)∆n,x
1 − Φ(x)
7
for all x ≥ 0 satisfying
x2 max EXi2 ≤ Bn2
1≤i≤n
and
∆n,x ≤ (1 + x)2 /A,
where |O(1)| ≤ A.
Theorem 5 provides a very general framework. The following result is a direct consequence of the
above general theorem.
Theorem 6 [Jing, Shao and Wang (2003)]. Let {an , n ≥ 1} be a sequence of positive numbers. Assume that
a2n ≤ Bn2 / max EXi2
(11)
1≤i≤n
and
∀ ε > 0,
Bn−2
n
X
EXi2 I{|Xi | > εBn /(1 + an )} → 0 as n → ∞.
(12)
i=1
Then
ln P (Sn /Vn ≥ x)
→1
ln(1 − Φ(x))
(13)
holds uniformly for x ∈ (0, an ).
When the Xi ’s have a finite (2 + δ)th moment for 0 < δ ≤ 1, we obtain (10) without assuming any
symmetric condition.
Theorem 7 [Jing, Shao and Wang (2003)]. Let 0 < δ ≤ 1 and set
Ln,δ =
n
X
1/(2+δ)
E|Xi |2+δ , dn,δ = Bn /Ln,δ
.
i=1
Then,
1 + x 2+δ
P (Sn /Vn ≥ x)
= 1 + O(1)
1 − Φ(x)
dn,δ
(14)
for 0 ≤ x ≤ dn,δ , where O(1) is bounded by an absolute constant. In particular, if dn,δ → ∞ as
n → ∞, we have
P (Sn ≥ xVn )
→1
(15)
1 − Φ(x)
uniformly in 0 ≤ x ≤ o(dn,δ ).
2
By the fact that 1 − Φ(x) ≤ 2e−x /2 /(1 + x) for x ≥ 0, it follows from (14) that the following
exponential non-uniform Berry-Esseen bound
|P (Sn /Vn ≥ x) − (1 − Φ(x))| ≤ A(1 + x)1+δ e−x
2 /2
/d2+δ
n,δ
(16)
holds for 0 ≤ x ≤ dn,δ .
Theorem 5 has been successfully applied to study the studentized bootstrap and the self-normalized
law of the iterated logarithm. The proof of Theorem 5 is quite complicated. Shao (2004) refined
Theorem 6 which only requires condition (12).
8
Theorem 8 [Shao (2004)] Let xn be a sequence of real numbers such that xn → ∞ and xn = o(Bn ).
Assume
n
X
−2
∀ ε > 0, Bn
EXi2 I{|Xi | > εBn /xn } → 0.
(17)
i=1
Then we have
ln P (Sn /Vn ≥ xn ) ∼ −x2n /2.
6
(18)
Self-normalized laws of the iterated logarithm
Let X, X1 , X2 , · · · be i.i.d. random variables. It is well-known that the Hartman-Wintner law of the
iterated logarithm holds if and only if the second moment of X is finite. In contrast to this classical
result, Griffin and Kuelbs (1989) established a self-normalized law of the iterated logarithm for all
distributions in the domain of attraction of a normal or stable law:
Theorem 9 [Griffin and Kuelbs (1989)]
(a) If EX = 0 and EX 2 I{|X| ≤ x} is slowly varying as x → ∞, then
lim sup
n→∞
Sn
=1
Vn (2 log log n)1/2
a.s.
(b) Under the conditions of Theorem 4, there is a positive constant C such that
lim sup
n→∞
Sn
=C
Vn (2 log log n)1/2
a.s.
By applying the self-normalized moderate deviation result of Theorem 4 and the subsequence
method, the precise constant C in Griffin and Kuelbs’s LIL can be determined.
Theorem 10 [Shao (1997)] Under the conditions of Theorem 4, we have
lim sup
n→∞
Sn
= (β(α, c1 , c2 ))1/2 a.s.
1/2
(log log n) Vn
For non-identically distributed independent random variables, as a direct consequence of Theorem
8, we have
Theorem 11 [Shao(2004)] Let X
P1n, X2 , · · ·2 be independent random variables with E(Xi ) = 0 and
2
2
E(Xi ) < ∞. Assume that Bn := i=1 EXi → ∞ as n → ∞ and that
∀ ε > 0, Bn−2
n
X
EXi2 I{|Xi | > εBn /(log log Bn )1/2 } → 0,
i=1
then
lim sup
n→∞
Sn
= 1 a.s.
Vn (2 log log Bn )1/2
9
7
Limiting distributions of self-normalized sums
Let X, X1 , X2 , · · · be i.i.d random variables. It is well-known that Sn /bn − an has a non-degenerate
limiting distribution for some suitably chosen real constants an and bn > 0 if and only if X is in the
domain of attraction of a stable law with index α (0 < α ≤ 2). When α = 2, it is equivalent to
EX 2 I(|X| ≤ x) is slowly varying as x → ∞, while for 0 < α < 2, it is equivalent to
(c1 + o(1))h(x)
(c2 + o(1))h(x)
, P (X ≤ −x) =
α
x
xα
as x → ∞, where c1 ≥ 0, c2 ≥ 0, c1 + c2 > 0 and h(x) is slowly varying at ∞. It is also known that
the normalizing constants an and bn are complicatedly determined by the slowly varying function l.
On the other hand, self-normalization shows again its neatness. Logan, Mallows, Rice and Shepp
(LMRS)(1973) proved that all limit laws of Sn /Vn,2 for X in the domain of attraction of a stable law
with index α ∈ (0, 2) have a subgaussian tail which depends in a complicated way on the parameter
α, and posed a conjecture which was solved in [47].
P (X ≥ x) =
Theorem 12 [Logan, Mallows, Rice and Shepp (1973) and Shao (1997)]
Under the conditions of Theorem 4, the limiting density function p(x) of Sn /Vn,2 exists and satisfies
as x → ∞
1 2 1/2 p
2
p(x) ∼
2β(α, c1 , c2 )e−x β(α,c1 ,c2 ) .
α π
Logan, Mallows, Rice and Shepp (1973) also stated that Sn /Vn is asymptotically normal if [and
perhaps only if] X is in the domain of attraction of the normal law and that it seems worthy of
conjecture that the only possible nontrivial limiting distributions of Sn /Vn are those obtained when
X follows a stable law. The first conjecture of LMRS was proved by Giné, Götze and Mason (1997)
and the second conjecture was recently confirmed by Chistyakov and Götze (2004).
Theorem 13 [Gine, Götze and Mason (1997)]
d.
Sn /Vn,2 −→ N (0, 1)
if and only if EX = 0 and EX 2 I{|X| ≤ x} is slowly varying.
Theorem 14 [Chistyakov and Götze (2004)]. Sn /Vn converges weakly to a random variable Z such
that P (|Z| = 1) < 1 if and only if
(i) X is in the domain of attraction of a stable law with index α ∈ (0, 2];
(ii) EX = 0 if 1 < α ≤ 2;
(iii) if α = 1, then X is in the domain of attraction of the Cauchy law and Feller’s condition holds,
that is, limn→∞ nE sin(X/an ) exists and is finite, where an = inf{x > 0 : nx−2 (1 + EX 2 I{|X| ≤
x}) ≤ 1}.
Chistyakov and Götze (2004) also proved that Sn /Vn converges weakly to Z with P (|Z| = 1) if
and only if P (|X| > x) is a slowly varying function at +∞.
For the asymptotic distribution of self-normalized censored sums and trimmed sums, we refer to
Hahn, Kuelbs and Weiner (1990) and Griffin and Pruitt (1991).
10
8
Weak invariance principle for self-normalized partial sum processes
Let X, X1 , X2 , · · · be i.i.d. random variables. As a generalization of the central limit theorem, the
classical weak invariance principle states that on an appropriate probability space,
1
1
sup | √ S[nt] − √ W (nt)| = o(1) in probability
nσ
n
0≤t≤1
if and only if EX = 0 and Var(X) = σ 2 , where {W (t), t ≥ 0} is a standard Wiener process. The
weak invariance principle is a stronger version of Donsker’s classical functional central limit theorem.
In view of the self-normalized central limit theorem, Csörgő, Szyszkowicz and Wang (2003) proved a
self-normalized version of the weak invariance principle.
Theorem 15 [Csörgő, Szyszkowicz and Wang (2003)]. As n → ∞, the following statements are
equivalent:
(a) EX = 0 and X is in the domain of attraction of the normal law;
(b) S[nt] /Vn → W (t) weakly on (D[0, 1], ρ), where ρ is the sup-norm metric for functions in D[0, 1],
and {W (t), 0 ≤ t ≤ 1} is a standard Wiener process;
(c) On an appropriate probability space, we can construct a standard Wiener process {W (t), t ≥ 0}
such that
√
sup |S[nt] /Vn − W (nt)/ n| = o(1) in probability.
0≤t≤1
The following are immediate corollaries of the above weak invariance principle. If EX = 0 and X
is in the domain of attraction of the normal law, then
P ( max Si /Vn ≤ x) → P ( sup W (t) ≤ x),
1≤i≤n
0≤t≤1
P ( max |Si |/Vn ≤ x) → P ( sup |W (t)| ≤ x),
1≤i≤n
P (n−1
P (n−1
0≤t≤1
n
X
i=1
n
X
Si2 /Vn2 ≤ x) → P
1
Z
W 2 (t)dt ≤ x ,
0
|Si |/Vn ≤ x) → P
Z
1
|W (t)|dt ≤ x .
0
i=1
When X is in the domain of attraction of a stable law, Hu and Shao (2005) obtained a similar
invariance principle.
Let N1 , N2 be two independent poisson random measures with intensity L on σ-finite measure
space (R2 , B 2 , L), where B 2 is the Borel σ-algebra of R2 and L is the Lebesgue measure on R2 . For
i = 1, 2 put
Ni (t, y) := Ni ([0, t] × [0, y]) for t ≥ 0, y ≥ 0
and define the following processes:
 −1 R ∞
−1−1/α
 α
R ∞ 0 Ni (t, y)y −2 dy R e
(Ni (t, y) − ty)y dy + 0 Ni (t, y)y −2 dy
∆α,i (t) =
 e−1 R ∞
−1−1/α dy
α
0 (Ni (t, y) − ty)y
11
if 0 < α < 1
if α = 1
if 1 < α ≤ 2
Theorem 16 [Hu and Shao (2005)] Assume that there exists α ∈ (0, 2), c1 ≥ 0, c2 ≥ 0, c1 + c2 > 0
and a slowly varying function h(x) such that
c1 + o(1)
c2 + o(1)
h(x) and P (X ≤ −x) =
h(x)
α
x
xα
as x → ∞ Moreover, assume that EX = 0 if 1 < α < 2 and X is symmetric if α = 1. Write
P (X ≥ x) =
∆α (t) = −ω1 ∆α,1 (t) + ω2 ∆α,2 (t),
e α (t) = ω12 ∆α/2,1 (t) + ω22 ∆α/2,2 (t), 0 ≤ t ≤ 1,
∆
where w1 =
c2
c1 +c2
1
α
, w2 =
c1
c1 +c2
1
α
, then in D[0, 1],
q
e α (1).
S[nt] /Vn ⇒ ∆α (t)/ ∆
Theorem 16 can be applied to obtain the limiting distribution of a unit-root test statistic when
the data is in the domain of attraction of a stable law.
9
Exponential inequalities for self-normalized processes
We have focused on limit theorems for self-normalized sums so far. Several papers have recently discussed moment inequalities for self-normalized processes. de la Pena, Klass and Lai (2004) established
very interesting exponential inequalities for general self-normalized processes.
Theorem 17 [de la Pena, Klass and Lai (2004)]. Let S and V be two variables with V ≥ 0 such that
λ2 2
V }≤1
2
(19)
S2
} ≤ 1.
2(V 2 + y 2 )
(20)
E exp{λS −
for all λ ∈ R. Then for all y > 0,
y
Ep
V 2 + y2
Consequently, if EV > 0, then
E exp
exp{
√
S2
≤ 2
4(V 2 + (EV )2 )
and
xS
E exp p
2
V + (EV
for x > 0. Moreover, for all p > 0,
E p
|S|
)2
p
V 2 + (EV )2
≤
√
2 exp(x2 )
1
≤ 2p+ 2 p Γ(p/2).
Condition (19) is satisfied
√ for a large classes of random variables (S, V ). Important examples
include: (i) S = WT , V = T , wherepWt is a standard Brownian motion, and T is a stopping time
with T < ∞ a.s.; (ii) S = Mt , V = qhMt i, where Mt is a continuous square-integrable martingale
Pn
P
2
with M0 = 0; (iii) S = ni=1 di , V =
i=1 di , where {di } is a sequence of variables adapted to an
increasing sequence of σ-fields {F i } and the di ’s are conditionally symmetric. They also developed
maximal inequalities and iterated logarithm bounds for self-normalized martingales.
12
10
Application to statistics
The idea of self-normalization is by no means new. Self-normalized statistics have been used, out
of necessity, for quite some time by now. The celebrated “Student t-statistic” tn , a classical object
familiar to virtually everyone who does data analysis, is closely related to Sn /Vn,2 via the following
identities
1/2
n−1
Sn
tn =
Vn,2 n − (Sn /Vn,2 )2
and
(
1/2 )
Sn
n
{tn ≥ t} =
≥t
.
Vn,2
n + t2 − 1
Thus, appropriate properties of self-normalized partial sums can be easily transformed to the tstatistic. For example, under the conditions of Theorem 3,
lim x−2
n ln P (tn ≥ xn ) = −
n→∞
1
2
√
for any xn → ∞ with xn = o( n).
The above result enables one to evaluate the exact Bahadur slope of all score tests for any univariate
model. He and Shao (1996) derived the exact Bahadur slopes of studentized score tests and showed
that under mild conditions the likelihood score is locally optimal in Bahadur efficiency. The selfnormalized technique was successfully applied in Chen and Shao (1997) to study the performance of
Monte Carlo methods for estimating ratios of normalizing constants.
Cao (2005) recently studied the moderate deviations for two-sample t-statistics.
Let X1 , · · · , Xn1 be a random sample from a population with mean µ1 and variance σ12 , and
Y1 , · · · , Yn2 be a random sample from another population with mean µ2 and variance σ22 . Define the
two sample t-statistic
X̄ − Ȳ − (µ1 − µ2 )
T = p 2
,
s1 /n1 + s22 /n2
where
n1
n2
1 X
1 X
2
2 2
s1 =
(Xi − X̄) , s2 =
(Yi − Ȳ )2 .
n1 − 1
n2 − 1
i=1
i=1
Theorem 18 [Cao(2005)] Let n = n1 + n2 . Assume that there are 0 < c1 ≤ c2 < ∞ such that
c1 ≤ n1 /n2 ≤ c2 . Then for any x := x(n1 , n2 ) satisfying x → ∞, x = o(n1/2 )
ln P (T ≥ x) ∼ −x2 /2
as n1 , n2 → ∞. If in addition, E|X1 |3 < ∞ and E|Y1 |3 < ∞, then
P (T ≥ x)
= 1 + O(1)(1 + x)3 n−1/2 d3
1 − Φ(x)
for 0 ≤ x ≤ n1/6 /d, where d3 = (E|X1 − µ1 |3 + E|Y1 − µ2 |3 )/(σ12 + σ22 )3/2 and O(1) is a finite constant
depending only on c1 and c2 . In particular,
P (T ≥ x)
→1
1 − Φ(x)
uniformly in x ∈ (0, o(n1/6 )).
For studentized U-statistic, we refer to Lai, Shao and Wang (2005).
13
References
[1] A. de Acosta, Large deviations for vector-valued functionals of a Markov chain: lower bounds.
Ann. Probab. 16 (1988), 925–960.
[2] R.R. Bahadur, Some limit theorems in statistics. Regional Conference Series in Appl. Math. No.
4, SIAM, Philadelphia, 1971.
[3] V. Bentkus and F. Götze, The Berry-Esséen bound for Student’s statistic. Ann. Probab. 24
(1996), 491-503.
[4] B. Bercu, E. Gassiat and E. Rio, Concentration inequalities, large and moderate deviations for
self-normalized empirical processes. Ann. Probab. 30 (2002), 1576–1604.
[5] H.Y. Cao, Moderate deviations for two sample t-statistics. Preprint.
[6] M.H. Chen and Q.M. Shao, On Monte Carlo methods for estimating ratios of normalizing constants. Ann. Statist. 25 (1997), 1563-1594.
[7] H. Chernoff, A measure of asymptotic efficiency for tests of a hypothesis based on the sum of
observations. Ann. Math. Statist. 23 (1952), 493-507.
[8] G.P. Chistyakov and F. Götze, Moderate deviations for self-normalized sums. Theory Probab. Appl.
47 (2003), 415–428.
[9] G.P. Chistyakov and F. Götze, Limit distributions of Studentized means. Ann. Probab. 32 (2004),
28-77.
[10] H. Cramér, Sur un nouveaux théoréme limite de la théorie des probabilités.
Indust No. 736 (1938), 5-23, Hermann, Paris.
Actualités Sci.
[11] S. Csörgő, E. Haeusler and D.M. Mason, The asymptotic distribution of trimmed sums. Ann.
Probab. 16 (1988), 672-699.
[12] S. Csörgő, E. Haeusler and D.E. Mason, The quantile-transform approach to the asymptotic
distribution of modulus trimmed sums. Sums, Trimmed Sums and Extremes (Hahn, M. G., Mason,
D.M., Weiner, D.C., eds) 337-354, Progr. Probab. 23 (1991), Birkhäuser, Boston.
[13] M. Csörgő and L. Horváth, Asymptotic representations of self-normalized sums. Probab. Math.
Statist. 9 (1988), 15–24.
[14] M. Csörgő and L. Horváth,
Sons, New York, 1993.
Weighted Approximations in Probability and Statistics. Wiley &
[15] M. Csörgő, B. Szyszkowicz and Q. Wang, Donsker’s theorem for self-normalized partial sums
processes. Ann. Probab. 31 (2003a), 1228-1240.
[16] M. Csörgő, B. Szyszkowicz and Q. Wang, Darling-Erdős theorems for self-normalized sums. Ann.
Probab. 31 (2003b), 676-692.
[17] H.E. Daniels, Saddlepoint approximations in statistics. Ann. Math. Statist. 25, 631-650 (1954).
[18] H.E. Daniels and G.A. Young, Saddlepoint approximation for the studentized mean, with an
application to the bootstrap. Biometrika 78, 169-179 (1991).
14
[19] D.A. Darling, The influence of the maximum term in the addition of independent random
variables. Trans. Amer. Math. Soc. 73 (1952), 95–107.
[20] A. Dembo and Q.M. Shao, Self-normalized large deviations in vector spaces. In: Progress in
Probability (Eberlein, Hahn, Talagrand, eds) Vol. 43 (1998a), 27-32
[21] A. Dembo and Q.M. Shao, Self-normalized moderate deviations and LILs Stochastic Process.
Appl. 75 (1998b), 51-65.
[22] A. Dembo and O. Zeitouni, Large Deviations Techniques and Applications. Jones and Bartlett,
Boston, 1992.
[23] M.D. Donsker and S.R.S Varadhan,
Stat. Physics 46 (1987), 1195–1232.
Large deviations for noninteracting particle systems.
J.
[24] E. Gine,, F. Götze and D.M. Mason, When is the Student t-statistic asymptotically standard
normal? Ann. Probab. 25 (1997), 1514-1531.
[25] P.S.Griffin and J. Kuelbs, Self-normalized laws of the iterated logarithm.
(1989), 1571–1601.
Ann. Probab. 17
[26] P.S. Griffin and D.M. Mason, On the asymptotic normality of self-normalized sums. Math. Proc.
camb. Phil. Soc. 109 (1991), 597-610.
[27] P.S. Griffin and W.E. Pruitt, Asymptotic normality and subsequential limits of trimmed sums.
Ann. Probab. 17 (1989), 1186-1219.
[28] P.S. Griffin and W.E. Pruitt, Weak convergence of trimmed sums. Sums, Trimmed Sums
and Extremes (Hahn, M. G., Mason, D.M., Weiner, D.C., eds) 55-80, Progr. Probab. 23 (1991),
Birkhäuser, Boston.
[29] M.G. Hahn, J. Kuelbs and D.C. Weiner, The asymptotic joint distribution of self-normalized
censored sums and sums of squares. Ann. Probab. 18 (1990), 1284-1341.
[30] M.G. Hahn, J. Kuelbs and D.C. Weiner, Asymptotic behavior of partial sums: a more robust
approach via trimming and self-normalization. Sums, Trimmed Sums and Extremes (Hahn, M.
G., Mason, D.M., Weiner, D.C., eds) 1-54, Progr. Probab. 23 (1991), Birkhäuser, Boston.
[31] X. He and Q.M. Shao, Bahadur efficiency and robustness of studentized score tests. Ann. Inst.
Statist. Math. 48 (1996), 295-314.
[32] Z.S. Hu and Q.M. Shao (2005). Self-normalized weak invariance principle in the domain of a
stable law (in preparation)
[33] J.L. Jensen, Saddlepoint Approximations. Oxford University Press, New York, 1995.
[34] B.Y. Jing, Q.M. Shao and Q.Y. Wang, Self-normalized Cramér type large deviations for independent random variables. Ann. Probab. (with B. Y. Jing and Q.Y. Wang) 31 (2003), 2167-2215.
[35] B.Y. Jing, Q.M. Shao and W. Zhou, Saddlepoint approximation for Student’s t-statistic with no
moment conditions. Ann. Statist. 32 (2004), 2679-2711.
15
[36] T.L. Lai, Q.M. Shao and Q.Y. Wang, Cramer type large deviations for studentized U-statistics
(in preparation)
[37] R. LePage, M. Woodroofe and J. Zinn, Convergence to a stable distribution via order statistics.
Ann. Probab. 9 (1981), 624-632.
[38] Yu. V. Linnik, Limit theorems for sums of independent random variables, taking account of large
deviations. Theory Probab. Appl. 7 (1962), 121-134.
[39] B.F.Logan, C.L. Mallows, S.O.Rice, and L.A. Shepp, Limit distributions of self-normalized sums.
Ann. Probab. 1 (1973), 788–809.
[40] R. Lugannani and S. Rice, Saddlepoint approximations for the distribution of the sum of independent random variables. Adv. Appl. Probab. 12, 475-490 (1980).
[41] R. Maller, Asymptotic normality of trimmed means in higher dimensions. Ann. Probab. 16
(1988), 1608-1622.
[42] R. Maller, A review of some asymptotic properties of trimmed sums of multivariate data. Sums,
Trimmed Sums and Extremes (Hahn, M. G., Mason, D.M., Weiner, D.C., eds) 179-214, Progr.
Probab. 23 (1991), Birkhäuser, Boston.
[43] V. de la Pena, M. Klass and T.Z. Lai, Self-normalized processes: exponential inequalities, moment
bounds and iterated logarithm laws. Ann. Probab. 32 (2004), 1902–1933.
[44] V.V. Petrov,
Sums of Independent Random Variables. Springer-Verlag, New York, 1975.
[45] Q.M. Shao, On a problem of Csörgő and Révész. Ann. Probab. 17 (1989), 809-812.
[46] Q.M. Shao, Cramér-type large deviation for Student’s t statistic. J. Theorect. Probab. 12, 387-398
(1999).
[47] Q.M. Shao, Self-normalized large deviations. Ann. Probab. 25 (1997), 285-328.
[48] V. Strassen, A converse to the law of the iterated logarithm. Z. Wahrsch. verw. Gebiete 4 (1966),
265–268.
[49] D.W. Stroock,
1984.
An Introduction to the Theory of Large Deviations. Springer-Verlag, Berlin,
[50] Q. Wang and B.Y. Jing, An exponential non-uniform Berry-Esseen bound for self-normalized
sums. Ann. Probab. 27, 2068-2088 (1999).
16