Download CH10.pdf

Ernesto Gutierrez-Miravete
Advanced Mathematics for
Engineering Applications
– An Introduction –
February 26, 2015
Springer
Contents
Part I Part Title
1
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.1 Mathematical Formulation of Problems in Engineering and Science
1.1.1 Differential Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.1.2 Variational Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Variables and Functions: Real and Complex . . . . . . . . . . . . . . . . . . . .
1.2.1 Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2.2 Variables: Real and Complex . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2.3 Functions of Real and Complex Variables . . . . . . . . . . . . . . . .
1.2.4 Elementary Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2.5 Derivatives and Integrals of Functions . . . . . . . . . . . . . . . . . . .
1.3 Vectors and Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3.1 Definition and Properties of Vectors, Matrices and
Determinants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3.2 Eigenvalues and Eigenvectors . . . . . . . . . . . . . . . . . . . . . . . . . .
1.3.3 Systems of Linear Algebraic Equations . . . . . . . . . . . . . . . . . .
1.3.4 Direct Solution of Systems of Linear Algebraic Equations . .
1.3.5 Iterative Solution of Systems of Linear Algebraic Equations
1.4 Interpolation and Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.4.1 Interpolation and Lagrange Polynomial . . . . . . . . . . . . . . . . . .
1.4.2 Cubic Spline Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.5 Numerical Differentiation and Integration . . . . . . . . . . . . . . . . . . . . . .
1.5.1 Numerical Differentiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.5.2 Numerical Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.6 Equations in a Single Variable and their Solution . . . . . . . . . . . . . . . .
1.6.1 Bisection Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.6.2 Fixed-Point Iteration Method . . . . . . . . . . . . . . . . . . . . . . . . . .
1.6.3 Newton-Raphson Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.6.4 Secant Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.6.5 Convergence Acceleration . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3
3
5
8
10
10
11
12
12
14
17
17
23
25
26
30
39
39
43
46
47
48
50
51
52
53
56
56
ix
x
Contents
1.7 Introduction to Ordinary Differential Equations . . . . . . . . . . . . . . . . .
1.7.1 First Order Ordinary Differential Equations . . . . . . . . . . . . . .
1.7.2 Higher Order Ordinary Differential Equations . . . . . . . . . . . .
1.7.3 General and Particular Solutions: Initial and Boundary
Value Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.8 Fundamentals of Probability and Statistics . . . . . . . . . . . . . . . . . . . . . .
1.8.1 Concepts of Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.8.2 Random Variables, Probability Mass and Density Functions
1.8.3 Cumulative Probability Distribution Function . . . . . . . . . . . . .
1.8.4 Expectation and Moment Generating Functions . . . . . . . . . . .
1.8.5 Law of Large Numbers and the Central Limit Theorem . . . .
1.8.6 Examples of Useful Probability Distributions . . . . . . . . . . . . .
1.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.10 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
57
58
59
60
61
61
62
63
63
65
65
80
81
2
Series Solutions of Differential Equations and Special Functions . . . . . 83
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
2.2 Power Series and their Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
2.3 Solving Differential Equations using Power Series . . . . . . . . . . . . . . . 87
2.4 The Method of Frobenius . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
2.5 Power Series Solutions Leading to Special Functions . . . . . . . . . . . . . 92
2.6 Bessel Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
2.7 Legendre Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
2.8 Application Examples of Special Functions . . . . . . . . . . . . . . . . . . . . . 105
2.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
2.10 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
3
Boundary Value Problems, Characteristic Functions and
Representations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
3.1 Boundary Value Problems and Characteristic Functions . . . . . . . . . . . 111
3.2 Some Examples of Characteristic Value Problems . . . . . . . . . . . . . . . 112
3.3 Orthogonality of Characteristic Functions . . . . . . . . . . . . . . . . . . . . . . 116
3.4 Characteristic Function Representations . . . . . . . . . . . . . . . . . . . . . . . . 118
3.5 Fourier Series Representations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
3.6 Fourier-Bessel Series Representations . . . . . . . . . . . . . . . . . . . . . . . . . 126
3.7 Legendre Series Representations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
3.8 Fourier Integral Representations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
3.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
3.10 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
4
Numerical Solution of Initial Value Problems
for Ordinary Differential Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
4.2 Fundamental Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
4.3 Single Step Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
Contents
4.4
4.5
4.6
4.7
4.8
xi
4.3.1 Euler’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
4.3.2 Higher Order Taylor Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 146
4.3.3 Runge-Kutta Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
4.3.4 Runge-Kutta-Fehlberg Method . . . . . . . . . . . . . . . . . . . . . . . . . 148
Multi step Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
4.4.1 Adams-Bashforth and Adams-Moulton Methods . . . . . . . . . . 150
4.4.2 Variable Step-Size Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
4.4.3 Richardson Extrapolation Method . . . . . . . . . . . . . . . . . . . . . . 152
Higher Order Equations and Systems of Equations . . . . . . . . . . . . . . . 156
Stiff Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
5
Numerical Solution of
Boundary Value Problems for Ordinary Differential Equations . . . . . . 167
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
5.2 Shooting Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
5.3 Finite Difference Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
5.4 Finite Volume Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
5.5 Approximation Methods based on Variational Formulations . . . . . . . 185
5.5.1 The classical Ritz Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
5.5.2 The Galerkin Finite Element Method . . . . . . . . . . . . . . . . . . . . 187
5.5.3 The Ritz Finite Element Method . . . . . . . . . . . . . . . . . . . . . . . . 197
5.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
5.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
6
Muti-variable Calculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
6.1 Functions of More than One independent Variable . . . . . . . . . . . . . . . 205
6.2 Spatial Differentiation of Vector Functions . . . . . . . . . . . . . . . . . . . . . 205
6.3 Line, Surface and Volume Integration of Vector Functions . . . . . . . . 212
6.4 Basic Concepts of Differential Geometry . . . . . . . . . . . . . . . . . . . . . . . 218
6.5 Introduction to Tensors and Tensor Analysis . . . . . . . . . . . . . . . . . . . . 223
6.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
6.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
7
Extrema, Solution of Nonlinear Systems of Algebraic Equations
and Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
7.2 Extrema of Functions of More than One Variable . . . . . . . . . . . . . . . . 235
7.3 Extrema of Functionals: Calculus of Variations . . . . . . . . . . . . . . . . . . 241
7.4 Numerical Solution of Systems of Nonlinear Algebraic Equations . . 246
7.4.1 Newton’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
7.4.2 Quasi-Newton Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
7.5 Numerical Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252
7.5.1 Steepest Descent Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
xii
Contents
7.5.2 Non-Gradient Numerical Optimization Methods . . . . . . . . . . 255
7.6 Linear Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
7.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
7.8 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
8
Analytical Methods for Initial and Boundary Value Problems in
Partial Differential Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
8.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
8.2 Partial Differential Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
8.2.1 First Order Quasi linear Equations . . . . . . . . . . . . . . . . . . . . . . 274
8.2.2 Second Order Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
8.2.3 Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
8.2.4 The Equations of Mathematical Physics . . . . . . . . . . . . . . . . . 278
8.3 Elliptic Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
8.3.1 Laplaces’s Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
8.3.2 The Method of Separation of Variables for Elliptic Problems 282
8.4 Parabolic Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
8.4.1 The Heat Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
8.4.2 The Method of Separation of Variables for Parabolic
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
8.5 Hyperbolic Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
8.5.1 The Wave Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332
8.5.2 The Method of Separation of Variables for Hyperbolic
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
8.6 Other Analytical Solution Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
8.6.1 Laplace Transform Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
8.6.2 The Method of Variation of Parameters . . . . . . . . . . . . . . . . . . 351
8.6.3 The Method of the Fourier Integral . . . . . . . . . . . . . . . . . . . . . 354
8.6.4 The Method of Green’s Functions . . . . . . . . . . . . . . . . . . . . . . 356
8.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361
8.8 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364
9
Numerical Methods for Initial and Boundary Value Problems in
Partial Differential Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367
9.2 Elliptic Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
9.2.1 Finite Difference Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
9.2.2 Finite Volume Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389
9.2.3 Finite Element Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392
9.3 Parabolic Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402
9.3.1 Finite Difference Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402
9.3.2 Finite Volume Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422
9.3.3 Finite Element Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423
9.4 Hyperbolic Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431
9.4.1 Finite Difference Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432
Contents
xiii
9.4.2 Finite Volume Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438
9.4.3 Finite Element Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 442
9.5 More Complex Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 448
9.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461
9.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466
10 Statistical and Probabilistic Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469
10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469
10.2 Generation of Pseudo-Random Numbers . . . . . . . . . . . . . . . . . . . . . . . 469
10.3 Generation of Random Variates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474
10.3.1 The Inverse Transform Method . . . . . . . . . . . . . . . . . . . . . . . . . 475
10.3.2 Inverse Transforms for the Normal and Log-Normal
Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475
10.3.3 Inverse Transforms for Empirical Distributions . . . . . . . . . . . 476
10.3.4 Inverse Transforms for Discrete Distributions . . . . . . . . . . . . . 485
10.4 Probabilistic and Statistical Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . 485
10.4.1 Stochastic Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485
10.4.2 Poisson Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485
10.4.3 Markov Chains and the Kolmogorov Balance Equations . . . . 488
10.4.4 Queueing Systems and Little’s Formula . . . . . . . . . . . . . . . . . 489
10.4.5 Matching Field Data with Theoretical Distributions . . . . . . . 491
10.4.6 Discrete Event Simulation Modeling . . . . . . . . . . . . . . . . . . . . 492
10.4.7 Reliability Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496
10.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 502
10.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503
Chapter 10
Statistical and Probabilistic Models
Abstract Probabilistic and statistical methods useful in applications are described
in this chapter. There are important for the solution of stochastic problems. Methods
for the generation of pseudo random mumbers and random variates are discussed.
Selected examples of stochastic models including Poisson processes, Markov chain,
queueing systems, discrete event simulation and reliability modeling are discussed.
10.1 Introduction
While the previous chapters focused on deterministic problems, stochastic phenomena are often encountered in applications. When the effect of variation can be safely
neglected, deterministic models are perfectly adequate. However, when the effect
of variation is important or when one is precisely interested in investigating those
effects, probabilistic and statistical methods must be employed.
From a modeling perspective, one may be interested in representing random variables through their probability density and distribution functions (as described in
Chapter 1), in simulating the variation itself using pseudo-random numbers, in examining the behavior of queueing, manufacturing, inventory and supply chain systems or in investigating the behavior of derived quantities such as the reliability,
hazard rate and mean time to failure. Modeling techniques based on the use of
pseudo-random number streams are generically known as Monte Carlo methods.
This chapter introduces, describes and explains various such probabilistic and statistical modeling methods useful in the study of stochastic systems.
10.2 Generation of Pseudo-Random Numbers
Recall that for a random variable X which is uniformly distributed in [0, 1] the uniform probability density function is
469
470
10 Statistical and Probabilistic Models
f (x) =
1, 0 ≤ x ≤ 1
0, otherwise
while its cumulative distribution function is

 0, x < 0
F(x) = x, 0 ≤ x < 1

1, x ≥ 1
A random number (RN) stream is a collection of uniformly distributed random
variables. A truly random stream of numbers has the following characteristics:
•
•
•
•
•
•
Uniformly distributed.
Continuous-valued.
E(R) = 12 .
1
σ 2 = 12
.
No autocorrelation between numbers.
No runs.
In practice one always works with streams of pseudo random numbers (PRN).
These have approximately the same characteristics as RN’s. PRN’s are generated
with a computer using a numerical algorithm embedded in a computer program
or routine. The requirements of a good PRNG routine include speed, portability,
long cycle, replicability and the ability of producing the PRNs with the desired
characteristics.
A robust and well established algorithm for PRN generation is the linear congruential method (LCM). More sophisticated approaches still use as foundation this
method. The fundamental relationship of the LCM is
Xi+1 = (aXi + c)mod(m)
This means that the value of Xi+1 is the remainder left from integer division of
aXi + c by m. Note that the values obtained form the LCM are from the set I =
{0, 1/m, 2/m, ..., (m − 1)/m}.
Figure 1 shows a sample of 10000 pseudo random numbers uniformly distributed
between zero and one obtained by one of the intrinsic widely available. While the
theoretical values of the mean and standard deviation of this distribution are 0.5 and
0.2986, respectively, the values for the sample are slightly different, namely 0.4984
and 0.2891, a difference below 1 %.
One key feature of the method is its period P (the number of numbers that can be
generated before the same number appears twice). It can be shown that the period is
related to the values of m and c as follows:

m = 2b ; |c| > 0
 m = 2b
b−2
P = m/4 = 2
m = 2b ; c = 0

b
m − 1 = 2 − 1 m = prime; c = 0
10.2 Generation of Pseudo-Random Numbers
471
1
’U01’
0.8
0.6
0.4
0.2
0
0
2000
4000
6000
Fig. 10.1 A sample of 10000 uniformly distributed pseudo-random numbers
8000
10000
472
10 Statistical and Probabilistic Models
Large simulations require large collections of PRNs and there is a need for still
longer periods. These can be obtained by the use of combined linear congruential
methods (CLCM).
Since in practice one always works with PRN streams it is necessary to check
how close are their characteristics to those of real RN streams. The key is to check
that the numbers are not related to each other. Assume a stream containing N PRN’s
has been produced. To verify their characteristics the stream is subjected to various
tests. In all cases, one states a hypothesis about a given characteristic of the stream
and then accepts it or rejects it with a given level of significance α where
α = P(rejectingH0 |H0 is true)
(i.e. Type I error).
In testing for uniformity The null hypothesis H0 is
Ri ∈ U[0, 1]
while the alternative hypothesis H1 is
Ri ∈
/ U[0, 1]
In testing for independence The null hypothesis H0 is
Ri ∈ independent
while the alternative hypothesis H1 is
Ri ∈
/ independent
Various tests are available.
In the Kolmogorov-Smirnov frequency test the pseudo-random numbers are first
arranged in increasing order
R1 < R2 < ... < RN
and the following variables are introduced
D+ = max(
i
− Ri )
N
D− = max(Ri −
i−1
)
N
in order to compute
D = max(D+ , D−)
10.2 Generation of Pseudo-Random Numbers
473
Once D has been computed, a critical value Dc is obtained from the K-S statistical
table for the desired α and the given N. Finally, if D > Dc , H0 is rejected and H1 is
accepted. If D ≤ Dc , H0 is not rejected (i.e. the numbers are uniformly distributed).
In the Chi-square frequency test the numbers are arranged instead into n classes
by subdividing the range [0, 1] into n subintervals and determining how many of the
numbers end up in each class i, (Oi ).
The test uses the statistic
n
(Oi − Ei )2
Ei
i=1
χ02 = ∑
where Ei = N/n are the expected numbers of numbers in each class for a uniform
distribution.
Once χ02 has been computed, a critical value χα2 ,n−1 is obtained from the Chisquare statistical table and if χ02 > χα2 ,n−1 , H0 is rejected (H1 is accepted). If χ02 ≤
χα2 ,n−1, H0 is not rejected (i.e. the numbers are assumed uniformly distributed).
The runs test aims to detect whether there are patterns in substrings of the stream.
One examines the stream and checks whether each number is followed by a larger
(+) or a smaller (−) number. Runs are the resulting patterns of +’s and −’s. In a
truly random sequence the mean and variance of the number of up and down runs a
are given by
µa =
2N − 1
3
and
σa2 =
16N − 29
90
When N > 20 the distribution of a is close to normal so the test statistic is
Z0 =
a − µa
σa
which has the normal distribution of mean zero and unit standard deviation (N(0, 1)).
Once Z0 has been computed a critical value zα /2 is obtained from the normal
statistical table. Finally, if Z0 < −zα /2 or Z0 > zα /2 , H0 is rejected (H1 is accepted).
If −zα /2 ≤ Z0 ≤ zα /2 , H0 is not rejected (i.e. the numbers are independent).
Other types of runs tests are also possible, for instance runs above and below the
mean and run lengths. For runs above and below the mean a test similar to the one
above is used but with the values of mean and variance for the number of runs b
µb =
and
2n1 n2 1
+
N
2
474
10 Statistical and Probabilistic Models
σb2 =
2n1 n2 (2n1 n2 − N)
N 2(N − 1)
where n1 and n2 are, respectively, the numbers of runs above and below the mean.
For run lengths one uses the Chi-square test to compare the observed number of
runs of given lengths against the expected number obtained in a truly independent
stream.
The autocorrelation test aims to detect correlation among numbers in the stream
separated by specific amount of numbers (lag). Consider the autocorrelation test for
a lag m. One investigates then the behavior of numbers Ri and Ri+ jm. If the autocorrelation ρim > 0 there is positive correlation (i.e. high numbers follow high numbers
and vice versa) and if ρim < 0 one has negative correlation. The autocorrelation is
estimated by
′
ρim
=
M
1
[ ∑ Ri+kmRi+(k+1)m] − 0.25
M + 1 k=0
where M is the largest integer satisfying i + (M + 1)m ≤ N. The test statistic is in
this case given by
Z0 =
′
ρim
′
σρim
where
σρim
′ =
√
13M + 7
12(M + 1)
Once Z0 has been computed, a critical value zα /2 is obtained from the normal
statistical table. Finally, if Z0 < −zα /2 or Z0 > zα /2 , H0 is rejected (H1 is accepted)
and if −zα /2 ≤ Z0 ≤ zα /2 , H0 is not rejected (i.e. the numbers can be regarded as
independent).
The gap test checks for independence by tracking down the pattern of gaps between a given digit in the stream. The test is performed using the KolmogorovSmirnov scheme and the poker test checks for independence based on the repetition
of certain digits in the sequence. The test is performed using the Chi-square scheme.
10.3 Generation of Random Variates
Stochastic simulation models are often useful when limited field data is available.
These models require as inputs the values of random variables with specified probability distributions. Such random variables are called random variates. Input data for
simulation models are collected from the field and/or produced from best available
estimates. However, the amount of data collected is rarely enough to run simulation
10.3 Generation of Random Variates
475
models and one must use the data to create PRN streams with statistical characteristics similar to those of the original data.
So, on the one hand one needs to identify the statistical characteristics of the original data and on the other one must be able to produce large collections of random
variates with statistical characteristics similar to those of the original data. Here we
focus on the second aspect, namely once we have determined the probability distribution applicable to our data we proceed to generate random variate streams for use
in the simulation. This is accomplished by the inverse transform method.
10.3.1 The Inverse Transform Method
The inverse transform method is an algorithm to produce strings of random variate values using strings of pseudo-random numbers as input. Given a random (or
pseudo-random) number R and a random variate to be produced X, one must first
determine a reasonable cumulative probability distribution function F(X) representative of X. The individual entries Ri , i = 1, 2, ..., n from the string of uniformly
distributed pseudo-random numbers are the assumed equal to values of F(X), i.e.
Fi (X) = Ri
Next, the equation F(X) = R is solved for X in terms of R, i.e.
Xi = Fi−1 (Ri )
The resulting values Xi , i = 1, 2, ..., n constitute the stream of random variates
Note that explicit, closed form expressions for the cumulative probability distribution function are needed in order to compute random variates by the inverse
transform method. However, the distribution function for the very frequently used
normal distribution cannot be expressed in terms of elementary functions and a numerical approximation (shown below) must be used. Also, empirical distributions
require special treatment.
10.3.2 Inverse Transforms for the Normal and Log-Normal
Distributions
The normal distribution does not have a closed-form inverse transformation. However, the following expression is an excellent approximation to the inverse cumulative distribution function of the standard normal distribution.
Xi ≈
R0.135
− (1 − Ri )0.135
i
0.1975
476
10 Statistical and Probabilistic Models
From the above, random variates with a normal distribution of mean µ and standard
deviation σ are readily obtained as
Xi ≈ µ + σ (
R0.135
− (1 − Ri )0.135
i
)
0.1975
A direct transformation can be used to produce two independent standard normal
variates Z1 and Z2 from two random numbers R1 and R2 according to
1
Z1 = (−2 lnR1 ) 2 cos(2π R2 )
and
1
Z2 = (−2 ln R1 ) 2 sin(2π R2 )
Normal random variates Xi with mean µ and standard deviation σ can then be obtained from
Xi = µ + σ Zi
If the random variable Y has the normal distribution with mean µ and variance
σ 2 , the associated random variable X = exp(Y ) has the lognormal distribution with
parameters µ and σ 2 .
Thus, random variates with a standard log-normal distribution can be generated
from the expression
Xi ≈ exp(
R0.135
− (1 − Ri )0.135
i
)
0.1975
Random variates with a log-normal distribution of parameters µ and σ are then
generated by
Xi ≈ exp[ µ + σ (
R0.135
− (1 − Ri )0.135
i
)]
0.1975
10.3.3 Inverse Transforms for Empirical Distributions
If no appropriate distribution can be found for the data one can resort to resampling
the data. This creates an empirical distribution. A simple empirical distribution can
be produced from given data by piecewise linear approximation.
Assume the available data points (observations) are arranged in increasing order
x1 , x2 , ..., xn. Assume also that a probability is assigned to each resulting range x j −
x j−1 such that the cumulative probability of the first j intervals is c j . The associated
random variate is obtained as
10.3 Generation of Random Variates
477
Xi = x j−1 +
x j − x j−1
(Ri − c j−1 )
c j − c j−1
when c j−1 < Ri ≤ c j .
The following table summarizes the inverse transform formulae used to obtain
random variate values of the variable X from some of the probability distribution
functions F most commonly used in simulation modeling work by pseudo-random
numbers R uniformly distributed ∈ [0, 1].
Distribution
Uniform
Exponential
Normal
Log-Normal
Inverse Transform
a + (b − a)R
− λ1 ln(1 − R)
µ + σ(R
0.135 −(1−R)0.135
)
0.1975
R0.135 −(1−R)0.135
)]
exp[ µ + σ (
0.1975
1
α [− ln(1 − Ri )] β
α − β ln(− ln(1 − R))
p
2
0 < R ≤ 12
a + pR(a − ab − ac + bc)
Triangular
b − R(ab − ac − b2 + bc) − ab + ac + b2 − bc 12 < R ≤ 1
Weibull
Gumbel
The following figures show the scatter patterns of 10000 random variates generated using pseudo-random number streams by the inverse transform method from
several of the distributions most frequently encountered in simulation modeling
work. For purposes of comparison, the distribution parameters have been selected
so that all the distributions have approximately the same measures of both mean and
central tendency.
As a comparison following table shows the theoretical values of the mean µ and
standard deviation σ of the selected distributions with the given parameters and
also the mean x̄ and standard deviation s values calculated from the 10000 data
points for each sample. The values shown suggest that the samples are reasonably
representative of their populations.
Distribution
µ
σ
x̄
s
Exponential 2.50000000 2.50000000 2.50886083 2.48171878
Weibull
3.01135828 1.50654796 3.01858187 1.50549877
Log-Normal 3.12699075 1.55472445 3.1326654 1.54815972
Gumbel
3.09274825 1.53695060 3.08671379 1.54223824
Uniform
3.0000000 1.44337567 2.99226618 1.44578135
Triangular 3.25000000 1.32680694 3.2573545 1.32822156
Normal
3.00000000 1.50000000 3.00684595 1.50091624
478
10 Statistical and Probabilistic Models
16
’Uniform’
14
12
10
8
6
4
2
0
0
2000
4000
6000
Fig. 10.2 A sample of 10000 random variates uniformly distributed between 5 and 5.5
8000
10000
10.3 Generation of Random Variates
479
16
’Exponential’
14
12
10
8
6
4
2
0
0
2000
4000
6000
Fig. 10.3 A sample of 10000 random variates exponentially distributed with parameter 0.4
8000
10000
480
10 Statistical and Probabilistic Models
16
’Weibull’
14
12
10
8
6
4
2
0
0
2000
4000
6000
8000
Fig. 10.4 A sample of 10000 random variates from a Weibull distribution with parameters 3.4, 2.1
10000
10.3 Generation of Random Variates
481
16
’LogNormal’
14
12
10
8
6
4
2
0
0
2000
4000
6000
8000
Fig. 10.5 A sample of 10000 random variates from a Log-Normal distribution with parameters
ln2.8, ln1.6
10000
482
10 Statistical and Probabilistic Models
16
’Gumbel’
14
12
10
8
6
4
2
0
0
2000
4000
6000
8000
Fig. 10.6 A sample of 10000 random variates from a Gumbel distribution with parameters 2.4, 1.2
10000
10.3 Generation of Random Variates
483
16
’Triangular’
14
12
10
8
6
4
2
0
0
2000
4000
6000
8000
Fig. 10.7 A sample of 10000 random variates from a triangular distribution with parameters
0, 3.26, 6.5
10000
484
10 Statistical and Probabilistic Models
16
’Normal’
14
12
10
8
6
4
2
0
0
2000
4000
6000
8000
Fig. 10.8 A sample of 10000 random variates from a normal distribution with parameters
0, 3.0, 1.5
10000
10.4 Probabilistic and Statistical Modeling
485
10.3.4 Inverse Transforms for Discrete Distributions
A similar procedure to the one indicated above can be used to produce discretely
distributed random variates. Since the cumulative distribution functions for discrete
distributions consist of discrete jumps separated by horizontal plateaus, lookup tables are a convenient and very efficient method of generating inverses.
10.4 Probabilistic and Statistical Modeling
Selected examples of important stochastic problems encountered in applications are
described and discussed in this section.
10.4.1 Stochastic Processes
A stochastic process takes place in a system when the state of the system changes
with time in a random manner. Many if not most natural and/or human-made processes are stochastic processes, although in some cases the random aspects can be
neglected.
A simple application of the probabilistic modeling approach is provided by the
random walk problem. Consider a random walker that decides to walk from a starting point marching step by step. However, to decide in which direction to go, before
taking each step the walker draws a pseudo-random number R from a uniform distribution between 0 and 1 and makes the decision based on the outcome as follows.
If 0 ≤ R < 0.25 he/she takes the step in the north direction, if 0.25 ≤ R < 0.5 he/she
takes the step in the south direction, if 0.5 ≤ R < 0.75 he/she takes the step in the
east direction, and if 0.75 ≤ R ≤ 1.0 he/she takes the step in the west direction.
10.4.2 Poisson Process
Often one is interested in the number of events which occur over a certain interval
of time, i.e. a counting process (N(t), t ≥ 0). A counting process is called a Poisson
process if it involves
• One arrival at a time.
• Random arrivals without rush or slack periods (stationary increments).
• Independent increments.
Under these circumstances, the probability that N(t) = n for t ≥ 0 and n =
0, 1, 2, ... is
486
10 Statistical and Probabilistic Models
40
’RWPath’
30
20
10
0
-10
-20
-30
-40
-50
-20
-10
0
10
20
Fig. 10.9 The path formed by 10000 steps of a random walker
30
40
50
60
70
10.4 Probabilistic and Statistical Modeling
487
300
’RandomWalkers’
200
100
0
-100
-200
-300
-300
-200
-100
0
Fig. 10.10 The final destination after 10000 steps for 5000 random walkers
100
200
300
488
10 Statistical and Probabilistic Models
P(N(t) = n) =
exp(−λ t)(λ t)n
n!
This means that N(t) has a Poisson distribution with parameter α = λ t. Its mean
and variance are E(N(t)) = V (N(t)) = α = λ t. It can be shown that if the number
of arrivals has a Poisson distribution, the inter arrival times have an exponential
distribution.
The random splitting property of Poisson processes states that if N(t) = N1 (t) +
N2 (t) is Poisson with rate λ , then N1 and N2 are independent Poisson with rates λ p
and λ (1 − p), where p and1 − p are the probabilities of the branches N1 and N2 .
Similarly, if N1 (t) + N2(t) = N(t), the reverse is true (random pooling property).
10.4.3 Markov Chains and the Kolmogorov Balance Equations
If the future probability characteristics of a system in which a stochastic process is
taking place depend only on the state of the system at the current time, one has a
Markov process or chain. The effect of the past on the future is contained in the
present state of the system.
As a simple example of a Markov chain consider a machine that works until it
fails (randomly) and then resumes work once is repaired. There are two states for
this system, namely
• The machine is busy (S0 )
• The machine is being repaired (S1 )
The system moves from state S0 to S1 at a rate λ and from S1 back to S0 at a rate µ .
As a second example consider now a facility where two machines A and B perform an operation. The machines fail randomly but resume work once they are repaired. The four possible states of this system are
•
•
•
•
Both machines are busy (S0 )
Machine A is being repaired while B is busy (S1 )
Machine B is being repaired while A is busy (S2 )
Both machines are being repaired (S3 )
Now λ1 and λ2 are, respectively, the failure rates of machines A and B while µ1 and
µ2 are the corresponding repair rates.
The Kolmogorov Balance Equations are differential equations relating the probabilities of the various states involved in a Markov chain P0 , P1, P2 and P3 . They are
obtained by a probability balance on the states.
For instance, for the second example above they are
dP0
= µ1 P1 + µ2 P2 − (λ1 + λ2 )P0
dt
10.4 Probabilistic and Statistical Modeling
489
dP1
= λ1 P0 + µ2 P3 − (µ1 + λ2 )P1
dt
dP2
= λ2 P0 + µ1 P3 − (λ1 + µ2 )P2
dt
dP3
= λ2 P1 + λ1 P2 − (µ1 + µ2 )P3
dt
Under steady state or equilibrium conditions the time derivatives are zero and the
probabilities are then related by a system of simultaneous linear algebraic equations.
10.4.4 Queueing Systems and Little’s Formula
A queueing system involves one or more servers which provide some service to
customers who arrive, line up and wait for service at a queue when all the servers
are busy. Typically, both arrival and service times are random variables. The single
server queue consists of a single server and a single queue. If the inter arrival times
of customers and the service times are exponentially distributed the resulting queue
is known as the M/M/1 queue.
Inter arrival and service times in queues are often modeled probabilistically. Two
examples of queueing systems are:
• The arrival of an operator in a machine shop mechanics at a centralized tool crib.
• The arrival of a customer at a bank.
Random inter arrival and service times are often simulated using exponential
distributions. However, sometimes a normal distribution or a truncated normal distribution may be more appropriate. Gamma and Weibull distributions are also used
to model arrival and service times.
A most important parameter of the queueing system is the server utilization ρ
defined as
ρ=
λ
µ
where λ is the mean arrival rate of customers from the outside world into the queueing system and µ is the mean service rate.
The single server queue can also be regarded as a Markov chain in which the
various states are distinguished only by the number of customers waiting in the
queue. Let us call the corresponding states S0 , S1, ..., Sn. The system can then move
into state Si either from Si−1 (if a new customer arrives before service is completed
for the customer being served) or from Si+1 if service is completed and the next
490
10 Statistical and Probabilistic Models
customer in line begins service before any new arrival. Let λi, j be the rate at which
the system transitions from state Si to state S j .
If the queue is at steady state, the Kolmogorov equations yield
Pn =
λn−1,n ...λ1,2λ0,1
P0
λn,n−1 ...λ2,1λ1,0
where Pn is the probability of encountering n customers in the system and λ0,1 = λ .
In investigating queueing systems one is also interested in other measures of system performance such as the expected number of customers in the system (queue
plus service station) L, the expected number of customers in the queue Lq , the expected wait time of customers in the system W , and the expected wait time of customers in the queue Wq . The above expectancies are related by Little’s Formula
L = λW
or that
Lq = λ Wq
A number of queueing problems have been solved yielding closed form expressions for the above performance parameters. For instance, for the M/M/1 queue at
steady state the results are as follows
ρ
λ
µ −λ = 1−ρ
1
1
µ −λ = µ (1−ρ )
2
ρ2
Lq µ (µλ −λ ) = 1−
ρ
ρ
Wq µ (µλ−λ ) = µ (1−
ρ)
Pn (1 − λµ )( λµ )n
L
W
Further, for the M/G/1 queue in which the service times have a mean of 1/ µ
and a variance σ 2 the corresponding results are
ρ 2 (1+σ 2 µ 2)
= ρ + 2(1−ρ )
2(1−ρ )
λ (1/µ 2 +σ 2 )
W µ1 + 2(1−ρ )
2
2
µ 2 +σ 2 )
σ 2 µ 2)
= ρ (1+
Lq λ (1/
2(1−ρ )
2(1−ρ )
µ 2 +σ 2 )
Wq λ (1/
2(1−ρ )
L
ρ+λ
P0 1 − ρ
2 (1/µ 2 +σ 2 )
10.4 Probabilistic and Statistical Modeling
491
10.4.5 Matching Field Data with Theoretical Distributions
A common situation in reliability analysis and modeling consists of finding probability distribution functions that best fit raw data known to consist of independent
and identically distributed (i.i.d.) variables. This is an optimization problem that is
solved using standard methods based on the notion of minimizing the squares of the
errors.
Time to failure of complex components is usually represented with the Weibull
distribution. If failures are completely random, the exponential distribution is used.
The log-normal distribution can also be used. For incomplete data uniform, triangular and beta distributions are used.
As with random numbers, field data must then be tested for independence. Some
useful tools are:
• Scatter Plots. Contiguous values in a string of values of a random variable are
plotted on a x − y plane. The resulting pattern of points is characteristic of the
distribution.
• Autocorrelation Plots. The covariance of values separated by a specified lag in
a string of values of a random variable is plotted as a function of the number of
data points.
• Runs Tests. This searches for peculiar patterns in substrings of numbers from a
larger stream.
Data must also be tested to see if they are Identically Distributed (Homogeneity
Tests). Some useful tools are:
•
•
•
•
•
•
•
Histograms.
Distribution Plots.
Quartile-Quartile Plots.
Kolmogorov-Smirnov.
Chi-squared.
Time Dependency of Distributions.
ANOVA.
The collected data typically consists of a limited number of data values. Simulation modeling require large numbers of multiple samples therefore the data must be
converted to a frequency distribution.
Raw data can be used as input for the simulation project but this is usually not
recommended except in special cases. More commonly, once data have been tested
for independence and correlation they are converted to a form suitable for use in the
simulation model. This is done by fitting it to some distribution. Once a distribution
fitting the data has been determined, input for the simulation program is produced
as random variates sampled from the fitted distribution. The frequency distribution
selected can be empirical or theoretical, discrete or continuous. Discrete distributions are rarely directly used. Instead, numerical values of discrete probabilities are
directly used. Effectively, continuous, theoretical distributions are almost always
employed.
492
10 Statistical and Probabilistic Models
Of the many available theoretical distributions 12 or so are commonly used in
simulation modeling. Data are fitted to theoretical distributions by identifying the
theoretical distribution which best represents the data. Note also that if the fitted
distribution is unbounded, values for simulation should be taken rather from a truncated version of the selected distribution in order to avoid unrealistic extreme values.
Each statistical distribution function has a physical basis. An understanding of
this basis is useful in determining candidate distributions to represent field data.
Following is a brief summary of the physical basis of selected distributions.
• Binomial. This represents the distribution of a random variable giving the number of successes in n independent trials each yielding either success or failure
with probabilities p and 1 − p, respectively.
• Geometric. This represents the distribution of a random variable giving the
number of independent trials required in an experiment before k successes are
achieved.
• Poisson. This represents the distribution of a random variable giving the number
of independent events occurring within a fixed amount of time.
• Normal. This represents the distribution of a random variable which is itself the
result of the sum of component processes.
• Lognormal. This represents the distribution of a random variable which is itself
the result of the product of component processes.
• Exponential. This represents the distribution of a random variable giving the
time interval between independent events.
• Gamma. A distribution of broad applicability restricted to non-negative random
variables.
• Beta. A distribution of broad applicability restricted to bounded random variables.
• Erlang. This represents the distribution of a random variable which is itself the
result of the sum of exponential component processes.
• Weibull. This represents the distribution of a random variable giving the time to
failure of a component.
• Uniform. This represents the distribution of a random variable whose values are
completely uncertain.
• Triangular. This represents the distribution of a random variable for which only
minimum, most likely and maximum values are known.
10.4.6 Discrete Event Simulation Modeling
Discrete event simulation (DES) models are used to simulate the time evolution
of systems with probabilistic inputs and outputs. In DES, the state of the system
changes at a discrete set of points in time based on the outcomes of random variate
generation processes.
Input data for DES models must often be created by inverse transform methods according to a specific statistical distribution. The required distribution must be
10.4 Probabilistic and Statistical Modeling
493
identified based on how well it represents collected data. Following is a brief summary of the real-life situations where the distributions mentioned above are likely to
be encountered.
• Binomial. Useful when there are only two possible outcomes of an experiment
which is repeated multiple times.
• Geometric. Useful also when there are only two possible outcomes of an experiment which is repeated multiple times.
• Poisson. Useful to represent the number of incoming customers or requests into
a system.
• Normal. Useful to represent the distribution of errors of all kinds.
• Lognormal. Useful for representation of times required to perform a given task
or accomplish a certain goal.
• Exponential. Useful to represent inter arrival times in all kinds of situations.
• Gamma. Useful also for representation of times required to perform a given task
or accomplish a certain goal but more general.
• Beta. Useful as a rough model under situation of ignorance and/or to represent
the proportion of non-conforming items in a set.
• Erlang. Useful to represent systems making simultaneous request for attention
from a server.
• Weibull. Useful to represent the life and/or reliability of components.
• Uniform. Useful when one knows nothing about the system.
• Triangular. Useful when one know little about the system.
As mentioned above, in discrete event simulation modeling the collected input
data is often replaced by computer generated random variate values which accurately represent the original data. Typically, several distributions may be considered
good candidates for this task. A histogram of the data can provide a first inkling
about the family of distribution function(s) which can well represent the data. Another simple test which can be used to quickly determine whether a given set of data
are adequately represented by a specific distribution is the construction of quantilequantile plots.
Assume that X is a random variable whose cumulative distribution function is
F. The q-quantile of X is that value γ of the random variable which satisfies the
equation
F(γ ) = P(X < γ ) = q
j(n+1)
In n data values are arranged in increasing order then the value of the k largest value will be denoted by g j/k . Therefore, the median is g1/2 , i.e. the n+1
2 -th
or half-way through the data set.
A common application of this concept is in investigating the distribution of income in a population where the total number of households is divided into five quintiles, (q = 0.2, 0.4, 0.6, 0.8 and 0.1) by increasing values of income. For instance, in
the USA, if one’s household income is more than about γ = 80, 000 dollars per year
then one belongs in the top quintile; i.e. one in five households is in that quintile.
494
10 Statistical and Probabilistic Models
Consider a collection of n values of the random variable X, xi for i = 1, 2, ..., n.
If the values are arranged according to their magnitude a new string of values is obtained which we call y j with j = 1, 2, ..., n. The new variable becomes immediately
an estimate for the ( j − 12 )/n quantile of X, i.e.
y j ≈ F −1 (
j − 12
)
n
Once an appropriate family of distributions has been selected one proceeds to determine the various distribution parameters using the collected data values. To determine the appropriateness of a given distribution in a particular situation goodnessof-fit tests are required. The tests verify the validity of the null hypothesis H0 which
states that the random variable X follows a specific distribution. Two commonly
used and easily implemented tests are the Chi-square test (applicable to large samples) and the Kolmogorov-Smirnov test (applicable to small samples and restricted
to continuous distributions).
For the Chi-square test the n data points are arranged into a desired number of
cells (k). The expected number of points to fall inside the i-th cell, Ei is then given
by
Ei = npi
where pi is the probability associated with that interval and is obtained from the
specified distribution.
For instance, consider the case of reliability data consisting of a total of n f failures, binned into cells representing number of failures within time intervals of uniform duration ∆ ti = ti+1 − ti . Assume that direct calculation of the failure rate (hazard function value) yields a reasonably constant trend and that an average value
is computed. Introduce then the null hypothesis that the data is exponentially distributed with constant failure rate λ̂ estimated as the average failure rate computed
from the data. The expected number of failures inside each time bin is then given by
Ei = n f × [exp(−λ̂ ti ) − exp(−λ̂ ti+1 )]
Next, using the actual number of data points contained in each cell, Oi one computes the statistic
n
(Oi − Ei )2
Ei
i=1
χ02 = ∑
To test the null hypothesis, the critical value of the statistic is determined as
χα2 ,k−s−1 where α is the confidence level and k − s − 1 is the number of degrees of
freedom and s is the number of parameters in the candidate distribution (s = 1 in the
case of the exponential distribution). Finally, if χ02 > χα2 ,k−s−1 then H0 is rejected
but if χ02 < χα2 ,k−s−1 the hypothesis cannot be rejected at the given confidence level.
10.4 Probabilistic and Statistical Modeling
495
If the null hypothesis cannot be rejected one can then calculate a confidence
interval for the distribution parameters. For instance, considering again the case of
the exponentially distributed reliability data above, one can show that a 100(1 −
α )% confidence interval for the value of λ̂ is given by
[λ̂ ×
χ02 (2n f , 1 − α /2)
χ 2 (2n f , α /2)
, λ̂ × 0
]
2n f
2n f
For the Kolmogorov-Smirnov test the n data points are also arranged in increasing order. If possible the data are made dimensionless dividing each value by the
largest value in the set. Then, one calculates the statistics
i
D+ = max( − Ri )
n
D− = max(Ri −
i−1
)
n
and
D = max(D+ , D−)
and compares D against the critical value Dc . When D < Dc the null hypothesis H0
cannot be rejected.
Sometimes input data for DES models is just not easily available. In those instances one must rely on related engineering data, expert opinion and physical or
other limitations to produce reasonable input values for the model. A few data values combined with the assumption of uniform, triangular or beta distribution can
provide a solid starting point for research.
In other situations, various inputs may be related to each other or the same input
quantity may exhibit autocorrelation over time and this fact needs to be taken into
account for proper simulation. Typical examples are in inventory modeling where
demand data affect lead time data and in stock trading where buy and sell orders
called to the broker tend to arrive in bursts.
When two correlated input variables X1 and X2 are involved one uses their covariance Cov(X1 , X2) or their correlation
ρ=
Cov(X1 , X2 )
σ2
σ1
If collected data values for the two values yields ρ >> 0 then one needs to generate
correlated variates.
The following algorithm generates two correlated random variates with normal
distributions with parameters µ1 , σ1 and µ2 , σ2 , respectively.
• Generate two independent standard normal variates Z1 and Z2 .
• Set X1 = µ1 + σ1 Z1
496
10 Statistical and Probabilistic Models
• Set X2 = µ2 + σ2 (ρ Z1 +
p
1 − ρ 2 Z2 )
10.4.7 Reliability Modeling
Reliability analysis is the investigation of the probability that a given system may
fail to perform as expected. In reliability modeling probabilistic simulation methods
are used to determine estimates of the useful lives of systems.
In reliability, the random variable of interest is the time to failure of an item T .
This time to failure may be calendar time or any other number indicative of the life
of the system.
The reliability function R(t) of an item is the probability that the failure time of
an item will exceed time t, i.e.
R(t) = Pr(T > t) = 1 − F(t)
where F(t) is the cumulative probability distribution function of the failure times,
i.e.
F(t) = Pr(T ≤ t) =
Z t
f (τ )d τ
0
representing the probability that the failure time of the item will be less that t. Here,
f (t) is the corresponding probability density function of failure time. In terms of the
density function, the reliability function is
R(t) = 1 −
Z t
f (τ )d τ =
0
Z ∞
f (τ )d τ
t
A very important quantity, the mean time to failure T̄ f of an item is then calculated as
T̄ f =
Z ∞
R(t)dt
0
The conditional probability that the item will fail in the time interval between t
and t + ∆ t given that it is still functioning at time t is given by
Pr(t < T ≤ t + ∆ t|T > t) =
Pr(t < T ≤ t + ∆ t) F(t + ∆ t) − F(t)
=
Pr(T > t)
R(t)
The failure rate or force of mortality or hazard rate function z(t) is a very important quantity since it measures the probability that an item that has survived till time
t will fail some time between t and t + ∆ t. Therefore, z(t) is given as
z(t) = lim
∆ tto0
f (t)
Pr(t ≤ T ≤ t + ∆ t|T > t)
1 F(t + ∆ t) − F(t)
= lim
=
∆t
∆ tto0 R(t)
∆t
R(t)
10.4 Probabilistic and Statistical Modeling
497
The four functions introduced above are all interrelated and knowledge of one of
them determines the other three.
The reliability of systems formed by various components is a function of the individual reliabilities of said components. A binary variable xi can be used to describe
whether a component i is working (1) or not (0). The structure function of a system
formed by multiple components is a simple binary variable that takes the value of 1
when the system is functioning and zero when it fails.
Components in systems may be arranged in series mode or in parallel mode. The
structure function φ of a system of n components arranged in series mode is simply
defined as
n
φ (xi ) = x1 x2 ...xn = ∏ xi
i=1
and that of the system where the components are arranged in parallel by
n
φ (xi ) = 1 − (1 − x1 )(1 − x2 )...(1 − xn) = 1 − ∏(1 − xi )
i=1
The corresponding system reliabilities are, for the system of n components in
series
n
Rs = R1 R2 ...Rn = ∏ Ri
i=1
and
n
n
R p = 1 − (1 − R1 )(1 − R2 )...(1 − Rn) = 1 − ∏(1 − Ri ) = 1 − ∏(Fi )
i=1
i=1
for the one with the components arranged in parallel. Here Ri , i = 1, 2, ..., n are the
reliabilities of the individual components.
More complex systems can be represented as composite arrangements of subsystems some arranged in series y some in parallel. Therefore, a system with n
components that still functions when at least k of components are still functioning
can be regarded as a collection of n series sub-systems arranged in parallel, each
containing k components. This arrangement is called a k-out-of-n structure
As an example consider an aircraft containing three engines but designed so that
it can continue flying even when one of the engines fails. When any of the engines
fails, the other two still work and the plane can keep flying.
As an example consider a system consisting of three components, 1,2,3 with
Weibull failure probability distribution functions given respectively as follows:
F1 = 1 − exp(−(
t 0.5
) )
10
498
10 Statistical and Probabilistic Models
F2 = 1 − exp(−
F3 = 1 − exp(−(
t
)
10
t 5
) )
10
If the components are arranged in series
Rs = R1 R2 R3 = (1 − F1 )(1 − F2 )(1 − F3 ) =
= exp(−0.3162t 0.5) exp(−0.1t) exp(−10−5t 5 )
The mean time to failure is calculated as T̄s = 3.4.
If one assumes instead that the components are connected in parallel
= 1 − (1 − exp(−0.3162t
0.5
R p = 1 − (1 − R1 )(1 − R2 )(1 − R3 ) =
))(1 − exp(−0.1t))(1 − exp(−10−5t 5 ))
The mean time to failure is T¯p = 27.19.
Finally for the 2-out-of-3 arrangement, the reliability is
R2oo3 = R1 R2 + R2 R3 + R3 R1 − 2R1 R2 R3
The value of T¯p = 8.58.
The figure below shows the values of the hazard function z(t) for the three cases
considered. For the series arrangement, the curve exhibits the often encountered
bathtub shape associated with an initial burn-in period, followed by a useful life
stage and then a wear out period. The other two curves are somewhat similar to
each other but the parallel arrangement exhibits the lowest values of z(t) while the
2-out-of-3 arrangement yields intermediate values but the shape of the curve resembles more the one corresponding to the parallel arrangement. Very similar results are obtained for this problem using the Monte Carlo method of simulating the
failures using random variates drawn from the given probability distributions and
sufficiently large samples.
As another example consider the case of deciding when best to replace an item
if its failure probability distribution function is known. The mean time between
replacements with replacement age t0 , T̄ )r is
T̄r =
Z t0
0
t f (t) + t0Pr(T ≥ t0 ) =
Z t0
0
(1 − F(t))dt
Note that if replacement only takes place when the item fails, the mean time between
replacements is the mean time to failure T̄ f .
Let F(t) be given by
F = 1 − exp(−(
t 3
) )
10
10.4 Probabilistic and Statistical Modeling
499
Fig. 10.11 The hazard function z(t) for three different arrangements of the three component system
discussed.
500
10 Statistical and Probabilistic Models
If the item is replaced before it fails, the cost incurred is c = 5. On the other hand,
if one waits until the item fails, one must incur additional (liability) expenses such
that the costs of replacement (following failure) is k = rc where r > 1 depending of
the severity of the consequences of failure. The objective is to determine an optimal
replacement age t0 for the component.
The mean total cost per replacement period c̄ is
c̄ = c + kPr(T, t0 ) = c + kF (t0 )
Alternatively, the total cost per replacement period if replacement takes place only
when the item fails is
c̄∞ = c + k
The mean total cost per unit time with replacement age t0 , C(t0) is given by
C̄ =
c̄
T̄r
If the item is replaced only when it fails, the mean cost per unit time is instead
given by
C̄∞ =
c̄∞
T̄ f
This quantity cannot be expressed in terms of elementary functions for the given
F(t) but it can be determined by symbolic manipulation or Monte Carlo simulation.
The figure below shows the graph of C̄ as a function of the replacement time t0 and
the coefficient r. The picture shows that for every value of r there is an optimal
replacement time t0 .
The optimal replacement time t0∗ is determined from the extremum condition
dC̄
=0
dt0
For instance, for r = 10, the resulting value of t0∗ = 3.69 and this yields C̄ = 2.04.
This should be compared against the corresponding cost resulting from replacing
only at failure which is C̄∞ = 6.16. Hence replacement at age t0∗ = 3.69 is cheaper
than waiting until the item fails.
When the liability costs are higher (say r = 100), the optimal replacement time
decreases to t0∗ = 1.71. However, the associated costs increases to C̄ = 4.38 since the
item must be replaced more frequently. Clearly, it may well happen that it is more
costly to replace before failure despite the additional cost of liability.
10.4 Probabilistic and Statistical Modeling
501
Fig. 10.12 The function C̄(t0 , r) representing the mean replacement cost per unit time for the case
discussed.
502
10 Statistical and Probabilistic Models
10.5 Exercises
1.- Use the linear congruential method to produce 10000 pseudo random numbers
uniformly distributed between 0 and 1. Examine the results.
2.- Use the inverse transform method to produce random variate values for random variables from the following distributions:
•
•
•
•
•
•
•
Uniform U(0, 6)
Triangular T (0, 3, 6)
Normal N(3, 1.5)
Exponential E(0.5)
Weibull W (3.5, 2)
Log-Normal LN(3, 1.5)
Gumbel G(0, 6)
Examine and compare the results.
3.- Obtain an actuarial life table and use the data in it to identify a probability
distribution that fits the data.
4.- A machine works continuously until it randomly fails. The machine is immediately sent for repair and resumes work once is repaired. Make a graph representing
the above Markov chain.
5.- At a facility, two machines perform an operation. The machines fail at random but resume work once they are repaired. If the failure and repair rates of the
machines are λ1 and λ2 and µ1 and µ2 , identify the possible states of this system are
and make a Markov chain graph of them.
6.- Derive the Kolmogorov equations for the single server queue.
7.- Derive Little’s formula relating L and W .
8.- Estimate the probability that a male in the US that has survived till his sixtieth birthday will pass away the day after his birthday. Then create a Monte Carlo
simulation model to estimate the probability.
9.- Consider a system consisting of three components, 1,2,3 with Weibull failure
probability distribution functions given respectively as follows:
F1 = 1 − exp(−(
t 0.5
) )
10
10.6 References
503
F2 = 1 − exp(−
F3 = 1 − exp(−(
t
)
10
t 5
) )
10
Assume the components can be arranged in series, in parallel or in a 2-out-of3 arrangement. Create Monte Carlo simulation models for the three different arrangement of this system and use them to estimate the corresponding hazard rate
functions.
10.- Consider the problem of determining the optimum replacement time t0∗ for
an item whose failure probability distribution function is given by
F = 1 − exp(−(
t 3
) )
10
If the item is replaced before it fails, the cost incurred is c = 5. On the other hand,
if one waits until the item fails, one must incur additional (liability) expenses such
that the costs of replacement (following failure) is k = 100c.
Create a Monte Carlo simulation model for this system and use it to estimate the
optimal replacement time.
10.6 References
1.- Banks, J. et al., Discrete-Event System Simulation, 3rd ed., Prentice-Hall, Englewood Cliffs, NJ, 2001.
2.- Cyganowski, S. et al., From Elementary Probability to Stochastic Differential
Equations with Maple, Springer, Berlin, 2002.
3.- Gnedenko, B.V. and Khinchin, A.Ya., An Elementary Introduction to the Theory of Probability, Dover, NY, 1962.
4.- Grimmett, G.R. and Stirzaker, D.R., Probability and Random Processes, 2nd.
ed., Clarendon Press, Oxford, 1995.
5.- Johnson, R.A. et al., Miller and Freund’s Probability and Statistics for Engineers, 8th ed., Pearson, Harlow, UK, 2014.
6.- Hogg, R.V. et al., Introduction to Mathematical Statistics, 6th ed., Pearson/PrenticeHall, Upper Saddle River, NJ, 2005.
504
10 Statistical and Probabilistic Models
7.- Kreyszig, E., Advanced Engineering Mathematics, 9th. ed., Wiley, NY, 2006.
8.- Rausand, M. and Hoyland, A., System Reliability Theory: Models, Statistical
Methods and Applications, Wiley, Hoboken, 2004.
9.- Ross, S.M., Introduction to Probability Models, 11th ed. Academic Press/Elsevier,
Amsterdam, 2014.
10.- Rubinstein, R.Y. and Kroese, D.P., Simulation and the Monte Carlo Method,
Wiley, Hoboken, 2008.
11.- Rychlik, I. and Ryden, J., Probability and Risk Analysis: An Introduction
for Engineers, Springer, Berlin, 2006.