Download Consider a collection of n independent non-linear birth

Estimating Rare Event Probabilities Using Truncated Saddlepoint Approximations
Timothy I. Matis, Ph.D. and Ivan G. Guardiola
Department of Industrial Engineering
Texas Tech University
P.O. Box 43061
Lubbock, Texas 79409-3061
Keywords: Rare events, saddlepoint approximations,
cumulant generating function
Abstract:
In this paper, we demonstrate how to estimate rare
event probabilities of convoluted random variables
using truncated saddlepoint approximations. This
quick analytical approach is surprisingly accurate in
the extreme tails, and may be used as an alternative to
importance sampling for robust estimation. Several
numerical examples, with the corresponding
Mathematica® code, are provided to demonstrate
implementation and highlight accuracy.
INTRODUCTION
Consider a collection of n independent random
variables { X 1 , X 2 ,, X n : X i  } that are not
necessarily identically distributed. We are interested
in estimating the rare event probability that the
convolution of these random variables exceeds some
large threshold. This may be mathematically stated as
estimating   Pr{S   } , where S 
n
X
i 1
i
and
   . In most practical models, the exact
computation of  is often intractable due to the
computational complexity of finding and evaluating
the convolution integral. Using simple Monte Carlo
techniques as a simulation based alternative, however,
is likewise inefficient due to the prohibitively large
number of trials required to obtain meaningful
estimates. Using importance sampling as an improved
Monte Carlo method has been shown to be a
computationally efficient simulation method.
Notwithstanding, this approach requires finding an
importance sampling density, which is not always
simple to do in practice. In particular, finding tilted
densities adds a layer of mathematical complexity and
computation, and using popular simulation-based
methods, such as Cross Entropy (Rubinstein and
Kroese 2004), adds a layer of computational effort. In
this paper, we propose an analytical approach to
efficiently approximating  using a truncated, as
opposed to full, saddlepoint approximation. This
truncated approach yields a compact approximate
expression for the pdf of the convolution that is easy to
integrate numerically and is surprisingly accurate in
estimating tail probabilities.
SADDLEPOINT APPROXIMATIONS
There are several moment-based methods that may
be used to approximate the probability distribution of
random variable. Most notably, the common Gaussian
approximations and Edgeworth expansions provide an
accurate representation of the central part of the
distribution, yet they often do not perform well in the
tails, with the latter method often providing negative
probability estimates. Saddlepoint approximations,
however, are powerful in that they provide accurate
estimates of tail probabilities. Though their derivation
is involved, especially for bivariate models (Wang
1990), the resulting formulas are easy to use. They
were seminally explored by Daniels (Daniels 1954),
and have received considerable recent attention in the
literature. Truncated saddlepoint approximations were
preliminarily explored by Renshaw (1999), and then
extended to bivariate models in (Renshaw 2000). The
truncated form of the saddlepoint has a simple
mathematical representation for both univariate and
bivariate models, and his numerical studies showed
that these approximations were often reasonably
accurate far into the tails at low levels of truncation.
The accuracy of truncated saddlepoint approxions in
the tail, coupled with the simple summative form of
the cumlant generating function for the convolution of
n
S   X i , makes this a mathematically amenable
i 1
and logical approach for approximating  .
Mathematical Development
Let K i ( ) be a cumulant generating function
(cgf) for the random variable X i , whose expanded
form is given by definition as

K i ( )  
i 1
k i i
,
i!
where k i denotes the ith cumulant of X i . Recall the
one-to-one correspondence of the cgf with a
probability distribution, and the direct correspondence
of the low order cumulants with the distribution
measures of expectation, variance, skewness, etc
(Kendall and Stuart 1967). Let K ( ) be the cumulant
generating function of S, from whence it follows that
n
K ( )   K i ( ) . Hence, finding an expression for
i 1
the convoluted cgf is almost immediate.
A saddlepoint approximation for a probability
distribution of S is given by
f ( x) 
Exp[ K ( 0 )   0 x]
,
2K ( 0 )
where  0 is the appropriate root of x  K ( 0 ) . Note
that a full saddlepoint approximation would entail
finding an expression for this root based on the full
mathematical representation of K ( ) . As noted
previously, however, this may yield a lengthy
expressions for the root  0 , which in turn makes the
subsequent numerical integration of the density
function difficult, if even possible. As an alternative,
consider an jth order truncated form of the cgf K ( ) ,
which we shall call K ( j ) ( ) , defined as
k i i
.
i!
i 1
Hence, the cumulants of K ( j ) ( ) are identical to
those of K ( ) up through the jth order, and are set to
zero above that. Substituting this truncated cgf into
the saddlepoint expression given previously, and
expanding all derivatives, yields the truncated
saddlepoint expression
 j ki

Exp i 0   o x 
 i 1 i!
,
f ( j ) ( x) 
i
j 2
k 
2  i  2 0
i!
i 0
k i 1 0i
where x  
. Hence, finding this root
i!
i 0
reduces to simply finding that for polynomial  0 . Note
j 1
that this expression is compact in terms of length and
may be implemented easily into any mathematical
software.
Comments on Truncated Saddlepoints
The accuracy of truncated saddlepoint
approximations in estimating rare event tail
probabilities will be investigated in the subsequent
section, but as a word of caution, note that the
denominator of the approximation may yield complex
numbers. As an example, the lower tail of the
approximation under third order truncation (j=3) is
susceptible to complexity. As such, it is possible that
the integral may not be able to span the entire range of
the integrating random variable, wherein appropriate
adjustments to the definition of the domain should be
made. While complexity is possible, however, it is not
given for many systems. In addition, the selection of
an “optimal” truncation level for a system is to some
extent dependent on the distributional characterization
system itself. Though the truncated saddlepoint
approximation converges to the full approximation in
the limit, higher order truncation is not always better
in a finite sense. As such, the saddlepoint
approximation should be evaluated repetitively at
multiple truncation levels for completeness if
computational resources allow.
j
K ( j ) ( )  
NUMERICAL EXAMPLES
Convoluted Bernoulli Distributions
Consider the collection of independent random
variables { X 1 , X 2 ,, X 20 : X i ~ Ber ( p  0.4)} ,
i.e. each random variable is distributed as a Bernoulli
with a constant probability of success p. Note that this
example is given in (Ross 2002) in the context of
importance sampling and the Mathematica code used
in this example is in Appendix A. Letting S 
20
 Xi ,
i 1
we are interested in estimating the rare event
probability   Pr{S  16} . It follows that the cgf
for X i is given by K i ( )  ln(. 4 * Exp[t ]  .6) , from
whence the cgf for S is given by
K ( )  ln (.4 * Exp[t ]  .6) 20 . The cumulants up to
the jth truncation order are extracted from
K ( ) through differentiation, i.e. note that


d i K ( )
ki 
d i
 0
. Letting j=3, these extracted
cumulant estimates are given by
0.15
0.125
0.1
0.075
0.05
0.025
5
10
15
20
Fig[1]: Truncated Saddlepoint Approximation for
Bernoulli Convolution
While the truncated saddlepoint method appears to
approximate the probability distribution of S well, our
primary interest is in using this to estimate the rare
event probability   Pr{S  16} . Hence,
k 2  4.8  Var[ S ],
numerically integrating the upper tail of f (3) ( x) from
x=16 through the loss of computer precision yields
ˆ  2.378x10 4 . This may be compared to the exact
k 3  0.96  SK [ S ]Var[ S ]3 / 2 .
solution, computed explicitly, of   3.17 x10 4 .
k1  8.0  E[ S ],
Now, finding the appropriate root  0 of
k i 1 0i
 02
x
k1  k 2 0  k 3
i!
2
i 0
j 1
 8.0  4.8 0  0.96
yields  0 
0.175
 02
2
 5  5  15  x
. Substituting this  0
10
into the truncated saddlepoint expression
 3 k i 0i

Exp
  o x
 i 1 i!

f ( 3) ( x ) 
i
1
k 
2  i  2 0
i!
i 0
yields an approximate probability density function for
S. Plotting this function over x  [0,20] yields the
following graph in Fig[1].
Convoluted Normal and Exponential
Distributions
Consider the collections of independent random
variables {X 1 , X 2 : X 1 ~ N (2,1), X 2 ~ Exp(1)} .
Note that the Mathematica code for this example is
given in Appendix B. Letting S  X 1  X 2 , we are
interested in estimating the rare event probability
  Pr{S  10} . It follows that
 Exp[2t  0.5t 2 
K ( )  ln 
 , from whence
1 t


k1  3  E[ S ],
k 2  2  Var[ S ],
k 3  2  SK [ S ]Var[ S ]3 / 2 ,
k 4  6, k 5  24, k 6  120, k 7  720,
k 8  5040, k 9  40320, k10  362880.
Using these extracted cumulants, the truncated
saddlepoint approximations f ( j ) ( x) for j  3, ,10
were specified and subsequently integrated
numerically from x=10 through the loss of precision
yielding the probability estimates of  in Tbl[1]. Note
that when specifying f ( j ) ( x) in practice, it is
important to choose the appropriate root  0 from the
set of j-1 roots for x 
k i 1 0i
, that one being such

i!
i 0
j 1
that the subsequent integration of the distribution is
real valued over the majority of the domain of
definition.
j
also have a practical interpretation whose form may be
specified recursively. The full development of this
procedure in a multiple convolution setting is future
extension to this work.
Though not commented on in this paper, another
interesting extension of this work would be to consider
the convolution of random variables from independent
stochastic processes. In particular, let each random
variable X i (t ) describe the state of a non-linear birth
death process at time t, and let S (t ) 
n
X
i 1
ˆ
-5
3
5.18x10
4
2.17x10-4
5
3.82x10-4
6
5.04x10-4
7
5.71x10-4
8
6.71x10-4
9
6.41x10-4
10
6.55x10-4
Tbl[1]: Truncated Saddlepoint Approximations of ˆ of
the jth order
As a basis for comparison, an importance sampling
estimate of this probability is given by ˆ  5.5 x10 4 .
CONCLUSIONS AND FUTURE RESEARCH
In this paper, we have shown that a truncated
saddlepoint representation of the probability
distribution of convoluted independent, though not
necessarily identically distributed, random variables is
a computationally amenable approach to estimating
rare event probabilities with a reasonable degree of
accuracy. While these estimates do not have a
corresponding confidence interval, or even an apparent
bound, they are simple to generate and are usually
accurate to at least the order of the true probability.
As such, this approach may be viewed as a quick
“first-pass” procedure in practice, or as an elegant
mathematical heuristic approach in theory.
While we have focused on estimating rare event
probabilities from a single convolution in this paper,
the extension to estimation from multiple convolutions
is relatively straightforward. In particular, the
truncated saddlepoint approximation is specified for a
multivariate distribution in Renshaw (2000). The
corresponding cross cumulants of the multivariate cgf
i
(t )
denote their convolution. The exact form of the cgf
K i ( ) is not known, yet the individual cumulants of
X i (t ) may be estimated using moment closure
methods (Matis 2000), from whence an approximate
representation of K i ( ) , and consequently K ( ) ,
may be specified. These may be used to likewise
specify an approximate probability distribution of
S (t ) whose subsequent integration yields probability
estimates for  t  Pr{S (t )   } . The accuracy of
these rare event probability estimates, given that the
cumulants themselves are now only estimates based on
moment closure, would be another interesting area of
study.
REFERENCES
Daniels, H.E., 1954. ”Saddlepoint
Approximations in Statistics.” Annals of
Mathematical Statistics, 25, pp. 631-650.
Kendall, M.G. and Stuart A., 1967. The Advanced
Theory of Statistics. Hafner, New York.
Matis, J.H. and Kiffe T.R., 2000. Stochastic
Population Models. Springer, New York.
Renshaw, E., 1998.”Saddlepoint approximations
for stochastic processes with truncated cumulant
generating functions.” IMA Journal of
mathematics Applied in Medicine and Biology,
15, pp. 41-52.
Renshaw, E., 2000. “Applying the Saddlepoint
Approximation to Bivariate Stochastic Processes.”
Mathematical Biosciences, 168, pp.57-75.
Ross, S.M., 2002. Simulation. Academic Press,
New York.
Rubenstein, R.V. and Kroese D.P., 2004. The
Cross-Entropy Method. Springer, New York.
Wang, S., 1990. “Saddlepoint Approximations for
Bivariate Distributions.” Journal of Applied
Probability, 27, pp. 586-597.
APPENDIX A
APPENDIX B