Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Ernesto Gutierrez-Miravete Advanced Mathematics for Engineering Applications – An Introduction – February 26, 2015 Springer Contents Part I Part Title 1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Mathematical Formulation of Problems in Engineering and Science 1.1.1 Differential Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.2 Variational Formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Variables and Functions: Real and Complex . . . . . . . . . . . . . . . . . . . . 1.2.1 Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.2 Variables: Real and Complex . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.3 Functions of Real and Complex Variables . . . . . . . . . . . . . . . . 1.2.4 Elementary Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.5 Derivatives and Integrals of Functions . . . . . . . . . . . . . . . . . . . 1.3 Vectors and Matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.1 Definition and Properties of Vectors, Matrices and Determinants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.2 Eigenvalues and Eigenvectors . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.3 Systems of Linear Algebraic Equations . . . . . . . . . . . . . . . . . . 1.3.4 Direct Solution of Systems of Linear Algebraic Equations . . 1.3.5 Iterative Solution of Systems of Linear Algebraic Equations 1.4 Interpolation and Approximation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.1 Interpolation and Lagrange Polynomial . . . . . . . . . . . . . . . . . . 1.4.2 Cubic Spline Interpolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Numerical Differentiation and Integration . . . . . . . . . . . . . . . . . . . . . . 1.5.1 Numerical Differentiation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5.2 Numerical Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6 Equations in a Single Variable and their Solution . . . . . . . . . . . . . . . . 1.6.1 Bisection Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6.2 Fixed-Point Iteration Method . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6.3 Newton-Raphson Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6.4 Secant Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6.5 Convergence Acceleration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 3 5 8 10 10 11 12 12 14 17 17 23 25 26 30 39 39 43 46 47 48 50 51 52 53 56 56 ix x Contents 1.7 Introduction to Ordinary Differential Equations . . . . . . . . . . . . . . . . . 1.7.1 First Order Ordinary Differential Equations . . . . . . . . . . . . . . 1.7.2 Higher Order Ordinary Differential Equations . . . . . . . . . . . . 1.7.3 General and Particular Solutions: Initial and Boundary Value Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.8 Fundamentals of Probability and Statistics . . . . . . . . . . . . . . . . . . . . . . 1.8.1 Concepts of Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.8.2 Random Variables, Probability Mass and Density Functions 1.8.3 Cumulative Probability Distribution Function . . . . . . . . . . . . . 1.8.4 Expectation and Moment Generating Functions . . . . . . . . . . . 1.8.5 Law of Large Numbers and the Central Limit Theorem . . . . 1.8.6 Examples of Useful Probability Distributions . . . . . . . . . . . . . 1.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.10 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 58 59 60 61 61 62 63 63 65 65 80 81 2 Series Solutions of Differential Equations and Special Functions . . . . . 83 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 2.2 Power Series and their Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 2.3 Solving Differential Equations using Power Series . . . . . . . . . . . . . . . 87 2.4 The Method of Frobenius . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 2.5 Power Series Solutions Leading to Special Functions . . . . . . . . . . . . . 92 2.6 Bessel Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 2.7 Legendre Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 2.8 Application Examples of Special Functions . . . . . . . . . . . . . . . . . . . . . 105 2.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 2.10 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 3 Boundary Value Problems, Characteristic Functions and Representations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 3.1 Boundary Value Problems and Characteristic Functions . . . . . . . . . . . 111 3.2 Some Examples of Characteristic Value Problems . . . . . . . . . . . . . . . 112 3.3 Orthogonality of Characteristic Functions . . . . . . . . . . . . . . . . . . . . . . 116 3.4 Characteristic Function Representations . . . . . . . . . . . . . . . . . . . . . . . . 118 3.5 Fourier Series Representations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 3.6 Fourier-Bessel Series Representations . . . . . . . . . . . . . . . . . . . . . . . . . 126 3.7 Legendre Series Representations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 3.8 Fourier Integral Representations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 3.9 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 3.10 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 4 Numerical Solution of Initial Value Problems for Ordinary Differential Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 4.2 Fundamental Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 4.3 Single Step Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Contents 4.4 4.5 4.6 4.7 4.8 xi 4.3.1 Euler’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 4.3.2 Higher Order Taylor Methods . . . . . . . . . . . . . . . . . . . . . . . . . . 146 4.3.3 Runge-Kutta Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 4.3.4 Runge-Kutta-Fehlberg Method . . . . . . . . . . . . . . . . . . . . . . . . . 148 Multi step Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 4.4.1 Adams-Bashforth and Adams-Moulton Methods . . . . . . . . . . 150 4.4.2 Variable Step-Size Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 4.4.3 Richardson Extrapolation Method . . . . . . . . . . . . . . . . . . . . . . 152 Higher Order Equations and Systems of Equations . . . . . . . . . . . . . . . 156 Stiff Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 5 Numerical Solution of Boundary Value Problems for Ordinary Differential Equations . . . . . . 167 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 5.2 Shooting Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 5.3 Finite Difference Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 5.4 Finite Volume Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182 5.5 Approximation Methods based on Variational Formulations . . . . . . . 185 5.5.1 The classical Ritz Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 5.5.2 The Galerkin Finite Element Method . . . . . . . . . . . . . . . . . . . . 187 5.5.3 The Ritz Finite Element Method . . . . . . . . . . . . . . . . . . . . . . . . 197 5.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 5.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204 6 Muti-variable Calculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205 6.1 Functions of More than One independent Variable . . . . . . . . . . . . . . . 205 6.2 Spatial Differentiation of Vector Functions . . . . . . . . . . . . . . . . . . . . . 205 6.3 Line, Surface and Volume Integration of Vector Functions . . . . . . . . 212 6.4 Basic Concepts of Differential Geometry . . . . . . . . . . . . . . . . . . . . . . . 218 6.5 Introduction to Tensors and Tensor Analysis . . . . . . . . . . . . . . . . . . . . 223 6.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 6.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 7 Extrema, Solution of Nonlinear Systems of Algebraic Equations and Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 7.2 Extrema of Functions of More than One Variable . . . . . . . . . . . . . . . . 235 7.3 Extrema of Functionals: Calculus of Variations . . . . . . . . . . . . . . . . . . 241 7.4 Numerical Solution of Systems of Nonlinear Algebraic Equations . . 246 7.4.1 Newton’s Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 7.4.2 Quasi-Newton Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251 7.5 Numerical Optimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 7.5.1 Steepest Descent Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254 xii Contents 7.5.2 Non-Gradient Numerical Optimization Methods . . . . . . . . . . 255 7.6 Linear Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 7.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 7.8 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 8 Analytical Methods for Initial and Boundary Value Problems in Partial Differential Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 8.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 8.2 Partial Differential Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274 8.2.1 First Order Quasi linear Equations . . . . . . . . . . . . . . . . . . . . . . 274 8.2.2 Second Order Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276 8.2.3 Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 8.2.4 The Equations of Mathematical Physics . . . . . . . . . . . . . . . . . 278 8.3 Elliptic Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280 8.3.1 Laplaces’s Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281 8.3.2 The Method of Separation of Variables for Elliptic Problems 282 8.4 Parabolic Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302 8.4.1 The Heat Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302 8.4.2 The Method of Separation of Variables for Parabolic Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303 8.5 Hyperbolic Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 8.5.1 The Wave Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332 8.5.2 The Method of Separation of Variables for Hyperbolic Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333 8.6 Other Analytical Solution Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345 8.6.1 Laplace Transform Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 345 8.6.2 The Method of Variation of Parameters . . . . . . . . . . . . . . . . . . 351 8.6.3 The Method of the Fourier Integral . . . . . . . . . . . . . . . . . . . . . 354 8.6.4 The Method of Green’s Functions . . . . . . . . . . . . . . . . . . . . . . 356 8.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361 8.8 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364 9 Numerical Methods for Initial and Boundary Value Problems in Partial Differential Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367 9.2 Elliptic Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369 9.2.1 Finite Difference Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369 9.2.2 Finite Volume Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389 9.2.3 Finite Element Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392 9.3 Parabolic Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402 9.3.1 Finite Difference Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402 9.3.2 Finite Volume Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422 9.3.3 Finite Element Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423 9.4 Hyperbolic Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431 9.4.1 Finite Difference Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432 Contents xiii 9.4.2 Finite Volume Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 438 9.4.3 Finite Element Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 442 9.5 More Complex Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 448 9.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461 9.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466 10 Statistical and Probabilistic Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469 10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 469 10.2 Generation of Pseudo-Random Numbers . . . . . . . . . . . . . . . . . . . . . . . 469 10.3 Generation of Random Variates . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474 10.3.1 The Inverse Transform Method . . . . . . . . . . . . . . . . . . . . . . . . . 475 10.3.2 Inverse Transforms for the Normal and Log-Normal Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 475 10.3.3 Inverse Transforms for Empirical Distributions . . . . . . . . . . . 476 10.3.4 Inverse Transforms for Discrete Distributions . . . . . . . . . . . . . 485 10.4 Probabilistic and Statistical Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . 485 10.4.1 Stochastic Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485 10.4.2 Poisson Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485 10.4.3 Markov Chains and the Kolmogorov Balance Equations . . . . 488 10.4.4 Queueing Systems and Little’s Formula . . . . . . . . . . . . . . . . . 489 10.4.5 Matching Field Data with Theoretical Distributions . . . . . . . 491 10.4.6 Discrete Event Simulation Modeling . . . . . . . . . . . . . . . . . . . . 492 10.4.7 Reliability Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 496 10.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 502 10.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 503 Chapter 10 Statistical and Probabilistic Models Abstract Probabilistic and statistical methods useful in applications are described in this chapter. There are important for the solution of stochastic problems. Methods for the generation of pseudo random mumbers and random variates are discussed. Selected examples of stochastic models including Poisson processes, Markov chain, queueing systems, discrete event simulation and reliability modeling are discussed. 10.1 Introduction While the previous chapters focused on deterministic problems, stochastic phenomena are often encountered in applications. When the effect of variation can be safely neglected, deterministic models are perfectly adequate. However, when the effect of variation is important or when one is precisely interested in investigating those effects, probabilistic and statistical methods must be employed. From a modeling perspective, one may be interested in representing random variables through their probability density and distribution functions (as described in Chapter 1), in simulating the variation itself using pseudo-random numbers, in examining the behavior of queueing, manufacturing, inventory and supply chain systems or in investigating the behavior of derived quantities such as the reliability, hazard rate and mean time to failure. Modeling techniques based on the use of pseudo-random number streams are generically known as Monte Carlo methods. This chapter introduces, describes and explains various such probabilistic and statistical modeling methods useful in the study of stochastic systems. 10.2 Generation of Pseudo-Random Numbers Recall that for a random variable X which is uniformly distributed in [0, 1] the uniform probability density function is 469 470 10 Statistical and Probabilistic Models f (x) = 1, 0 ≤ x ≤ 1 0, otherwise while its cumulative distribution function is 0, x < 0 F(x) = x, 0 ≤ x < 1 1, x ≥ 1 A random number (RN) stream is a collection of uniformly distributed random variables. A truly random stream of numbers has the following characteristics: • • • • • • Uniformly distributed. Continuous-valued. E(R) = 12 . 1 σ 2 = 12 . No autocorrelation between numbers. No runs. In practice one always works with streams of pseudo random numbers (PRN). These have approximately the same characteristics as RN’s. PRN’s are generated with a computer using a numerical algorithm embedded in a computer program or routine. The requirements of a good PRNG routine include speed, portability, long cycle, replicability and the ability of producing the PRNs with the desired characteristics. A robust and well established algorithm for PRN generation is the linear congruential method (LCM). More sophisticated approaches still use as foundation this method. The fundamental relationship of the LCM is Xi+1 = (aXi + c)mod(m) This means that the value of Xi+1 is the remainder left from integer division of aXi + c by m. Note that the values obtained form the LCM are from the set I = {0, 1/m, 2/m, ..., (m − 1)/m}. Figure 1 shows a sample of 10000 pseudo random numbers uniformly distributed between zero and one obtained by one of the intrinsic widely available. While the theoretical values of the mean and standard deviation of this distribution are 0.5 and 0.2986, respectively, the values for the sample are slightly different, namely 0.4984 and 0.2891, a difference below 1 %. One key feature of the method is its period P (the number of numbers that can be generated before the same number appears twice). It can be shown that the period is related to the values of m and c as follows: m = 2b ; |c| > 0 m = 2b b−2 P = m/4 = 2 m = 2b ; c = 0 b m − 1 = 2 − 1 m = prime; c = 0 10.2 Generation of Pseudo-Random Numbers 471 1 ’U01’ 0.8 0.6 0.4 0.2 0 0 2000 4000 6000 Fig. 10.1 A sample of 10000 uniformly distributed pseudo-random numbers 8000 10000 472 10 Statistical and Probabilistic Models Large simulations require large collections of PRNs and there is a need for still longer periods. These can be obtained by the use of combined linear congruential methods (CLCM). Since in practice one always works with PRN streams it is necessary to check how close are their characteristics to those of real RN streams. The key is to check that the numbers are not related to each other. Assume a stream containing N PRN’s has been produced. To verify their characteristics the stream is subjected to various tests. In all cases, one states a hypothesis about a given characteristic of the stream and then accepts it or rejects it with a given level of significance α where α = P(rejectingH0 |H0 is true) (i.e. Type I error). In testing for uniformity The null hypothesis H0 is Ri ∈ U[0, 1] while the alternative hypothesis H1 is Ri ∈ / U[0, 1] In testing for independence The null hypothesis H0 is Ri ∈ independent while the alternative hypothesis H1 is Ri ∈ / independent Various tests are available. In the Kolmogorov-Smirnov frequency test the pseudo-random numbers are first arranged in increasing order R1 < R2 < ... < RN and the following variables are introduced D+ = max( i − Ri ) N D− = max(Ri − i−1 ) N in order to compute D = max(D+ , D−) 10.2 Generation of Pseudo-Random Numbers 473 Once D has been computed, a critical value Dc is obtained from the K-S statistical table for the desired α and the given N. Finally, if D > Dc , H0 is rejected and H1 is accepted. If D ≤ Dc , H0 is not rejected (i.e. the numbers are uniformly distributed). In the Chi-square frequency test the numbers are arranged instead into n classes by subdividing the range [0, 1] into n subintervals and determining how many of the numbers end up in each class i, (Oi ). The test uses the statistic n (Oi − Ei )2 Ei i=1 χ02 = ∑ where Ei = N/n are the expected numbers of numbers in each class for a uniform distribution. Once χ02 has been computed, a critical value χα2 ,n−1 is obtained from the Chisquare statistical table and if χ02 > χα2 ,n−1 , H0 is rejected (H1 is accepted). If χ02 ≤ χα2 ,n−1, H0 is not rejected (i.e. the numbers are assumed uniformly distributed). The runs test aims to detect whether there are patterns in substrings of the stream. One examines the stream and checks whether each number is followed by a larger (+) or a smaller (−) number. Runs are the resulting patterns of +’s and −’s. In a truly random sequence the mean and variance of the number of up and down runs a are given by µa = 2N − 1 3 and σa2 = 16N − 29 90 When N > 20 the distribution of a is close to normal so the test statistic is Z0 = a − µa σa which has the normal distribution of mean zero and unit standard deviation (N(0, 1)). Once Z0 has been computed a critical value zα /2 is obtained from the normal statistical table. Finally, if Z0 < −zα /2 or Z0 > zα /2 , H0 is rejected (H1 is accepted). If −zα /2 ≤ Z0 ≤ zα /2 , H0 is not rejected (i.e. the numbers are independent). Other types of runs tests are also possible, for instance runs above and below the mean and run lengths. For runs above and below the mean a test similar to the one above is used but with the values of mean and variance for the number of runs b µb = and 2n1 n2 1 + N 2 474 10 Statistical and Probabilistic Models σb2 = 2n1 n2 (2n1 n2 − N) N 2(N − 1) where n1 and n2 are, respectively, the numbers of runs above and below the mean. For run lengths one uses the Chi-square test to compare the observed number of runs of given lengths against the expected number obtained in a truly independent stream. The autocorrelation test aims to detect correlation among numbers in the stream separated by specific amount of numbers (lag). Consider the autocorrelation test for a lag m. One investigates then the behavior of numbers Ri and Ri+ jm. If the autocorrelation ρim > 0 there is positive correlation (i.e. high numbers follow high numbers and vice versa) and if ρim < 0 one has negative correlation. The autocorrelation is estimated by ′ ρim = M 1 [ ∑ Ri+kmRi+(k+1)m] − 0.25 M + 1 k=0 where M is the largest integer satisfying i + (M + 1)m ≤ N. The test statistic is in this case given by Z0 = ′ ρim ′ σρim where σρim ′ = √ 13M + 7 12(M + 1) Once Z0 has been computed, a critical value zα /2 is obtained from the normal statistical table. Finally, if Z0 < −zα /2 or Z0 > zα /2 , H0 is rejected (H1 is accepted) and if −zα /2 ≤ Z0 ≤ zα /2 , H0 is not rejected (i.e. the numbers can be regarded as independent). The gap test checks for independence by tracking down the pattern of gaps between a given digit in the stream. The test is performed using the KolmogorovSmirnov scheme and the poker test checks for independence based on the repetition of certain digits in the sequence. The test is performed using the Chi-square scheme. 10.3 Generation of Random Variates Stochastic simulation models are often useful when limited field data is available. These models require as inputs the values of random variables with specified probability distributions. Such random variables are called random variates. Input data for simulation models are collected from the field and/or produced from best available estimates. However, the amount of data collected is rarely enough to run simulation 10.3 Generation of Random Variates 475 models and one must use the data to create PRN streams with statistical characteristics similar to those of the original data. So, on the one hand one needs to identify the statistical characteristics of the original data and on the other one must be able to produce large collections of random variates with statistical characteristics similar to those of the original data. Here we focus on the second aspect, namely once we have determined the probability distribution applicable to our data we proceed to generate random variate streams for use in the simulation. This is accomplished by the inverse transform method. 10.3.1 The Inverse Transform Method The inverse transform method is an algorithm to produce strings of random variate values using strings of pseudo-random numbers as input. Given a random (or pseudo-random) number R and a random variate to be produced X, one must first determine a reasonable cumulative probability distribution function F(X) representative of X. The individual entries Ri , i = 1, 2, ..., n from the string of uniformly distributed pseudo-random numbers are the assumed equal to values of F(X), i.e. Fi (X) = Ri Next, the equation F(X) = R is solved for X in terms of R, i.e. Xi = Fi−1 (Ri ) The resulting values Xi , i = 1, 2, ..., n constitute the stream of random variates Note that explicit, closed form expressions for the cumulative probability distribution function are needed in order to compute random variates by the inverse transform method. However, the distribution function for the very frequently used normal distribution cannot be expressed in terms of elementary functions and a numerical approximation (shown below) must be used. Also, empirical distributions require special treatment. 10.3.2 Inverse Transforms for the Normal and Log-Normal Distributions The normal distribution does not have a closed-form inverse transformation. However, the following expression is an excellent approximation to the inverse cumulative distribution function of the standard normal distribution. Xi ≈ R0.135 − (1 − Ri )0.135 i 0.1975 476 10 Statistical and Probabilistic Models From the above, random variates with a normal distribution of mean µ and standard deviation σ are readily obtained as Xi ≈ µ + σ ( R0.135 − (1 − Ri )0.135 i ) 0.1975 A direct transformation can be used to produce two independent standard normal variates Z1 and Z2 from two random numbers R1 and R2 according to 1 Z1 = (−2 lnR1 ) 2 cos(2π R2 ) and 1 Z2 = (−2 ln R1 ) 2 sin(2π R2 ) Normal random variates Xi with mean µ and standard deviation σ can then be obtained from Xi = µ + σ Zi If the random variable Y has the normal distribution with mean µ and variance σ 2 , the associated random variable X = exp(Y ) has the lognormal distribution with parameters µ and σ 2 . Thus, random variates with a standard log-normal distribution can be generated from the expression Xi ≈ exp( R0.135 − (1 − Ri )0.135 i ) 0.1975 Random variates with a log-normal distribution of parameters µ and σ are then generated by Xi ≈ exp[ µ + σ ( R0.135 − (1 − Ri )0.135 i )] 0.1975 10.3.3 Inverse Transforms for Empirical Distributions If no appropriate distribution can be found for the data one can resort to resampling the data. This creates an empirical distribution. A simple empirical distribution can be produced from given data by piecewise linear approximation. Assume the available data points (observations) are arranged in increasing order x1 , x2 , ..., xn. Assume also that a probability is assigned to each resulting range x j − x j−1 such that the cumulative probability of the first j intervals is c j . The associated random variate is obtained as 10.3 Generation of Random Variates 477 Xi = x j−1 + x j − x j−1 (Ri − c j−1 ) c j − c j−1 when c j−1 < Ri ≤ c j . The following table summarizes the inverse transform formulae used to obtain random variate values of the variable X from some of the probability distribution functions F most commonly used in simulation modeling work by pseudo-random numbers R uniformly distributed ∈ [0, 1]. Distribution Uniform Exponential Normal Log-Normal Inverse Transform a + (b − a)R − λ1 ln(1 − R) µ + σ(R 0.135 −(1−R)0.135 ) 0.1975 R0.135 −(1−R)0.135 )] exp[ µ + σ ( 0.1975 1 α [− ln(1 − Ri )] β α − β ln(− ln(1 − R)) p 2 0 < R ≤ 12 a + pR(a − ab − ac + bc) Triangular b − R(ab − ac − b2 + bc) − ab + ac + b2 − bc 12 < R ≤ 1 Weibull Gumbel The following figures show the scatter patterns of 10000 random variates generated using pseudo-random number streams by the inverse transform method from several of the distributions most frequently encountered in simulation modeling work. For purposes of comparison, the distribution parameters have been selected so that all the distributions have approximately the same measures of both mean and central tendency. As a comparison following table shows the theoretical values of the mean µ and standard deviation σ of the selected distributions with the given parameters and also the mean x̄ and standard deviation s values calculated from the 10000 data points for each sample. The values shown suggest that the samples are reasonably representative of their populations. Distribution µ σ x̄ s Exponential 2.50000000 2.50000000 2.50886083 2.48171878 Weibull 3.01135828 1.50654796 3.01858187 1.50549877 Log-Normal 3.12699075 1.55472445 3.1326654 1.54815972 Gumbel 3.09274825 1.53695060 3.08671379 1.54223824 Uniform 3.0000000 1.44337567 2.99226618 1.44578135 Triangular 3.25000000 1.32680694 3.2573545 1.32822156 Normal 3.00000000 1.50000000 3.00684595 1.50091624 478 10 Statistical and Probabilistic Models 16 ’Uniform’ 14 12 10 8 6 4 2 0 0 2000 4000 6000 Fig. 10.2 A sample of 10000 random variates uniformly distributed between 5 and 5.5 8000 10000 10.3 Generation of Random Variates 479 16 ’Exponential’ 14 12 10 8 6 4 2 0 0 2000 4000 6000 Fig. 10.3 A sample of 10000 random variates exponentially distributed with parameter 0.4 8000 10000 480 10 Statistical and Probabilistic Models 16 ’Weibull’ 14 12 10 8 6 4 2 0 0 2000 4000 6000 8000 Fig. 10.4 A sample of 10000 random variates from a Weibull distribution with parameters 3.4, 2.1 10000 10.3 Generation of Random Variates 481 16 ’LogNormal’ 14 12 10 8 6 4 2 0 0 2000 4000 6000 8000 Fig. 10.5 A sample of 10000 random variates from a Log-Normal distribution with parameters ln2.8, ln1.6 10000 482 10 Statistical and Probabilistic Models 16 ’Gumbel’ 14 12 10 8 6 4 2 0 0 2000 4000 6000 8000 Fig. 10.6 A sample of 10000 random variates from a Gumbel distribution with parameters 2.4, 1.2 10000 10.3 Generation of Random Variates 483 16 ’Triangular’ 14 12 10 8 6 4 2 0 0 2000 4000 6000 8000 Fig. 10.7 A sample of 10000 random variates from a triangular distribution with parameters 0, 3.26, 6.5 10000 484 10 Statistical and Probabilistic Models 16 ’Normal’ 14 12 10 8 6 4 2 0 0 2000 4000 6000 8000 Fig. 10.8 A sample of 10000 random variates from a normal distribution with parameters 0, 3.0, 1.5 10000 10.4 Probabilistic and Statistical Modeling 485 10.3.4 Inverse Transforms for Discrete Distributions A similar procedure to the one indicated above can be used to produce discretely distributed random variates. Since the cumulative distribution functions for discrete distributions consist of discrete jumps separated by horizontal plateaus, lookup tables are a convenient and very efficient method of generating inverses. 10.4 Probabilistic and Statistical Modeling Selected examples of important stochastic problems encountered in applications are described and discussed in this section. 10.4.1 Stochastic Processes A stochastic process takes place in a system when the state of the system changes with time in a random manner. Many if not most natural and/or human-made processes are stochastic processes, although in some cases the random aspects can be neglected. A simple application of the probabilistic modeling approach is provided by the random walk problem. Consider a random walker that decides to walk from a starting point marching step by step. However, to decide in which direction to go, before taking each step the walker draws a pseudo-random number R from a uniform distribution between 0 and 1 and makes the decision based on the outcome as follows. If 0 ≤ R < 0.25 he/she takes the step in the north direction, if 0.25 ≤ R < 0.5 he/she takes the step in the south direction, if 0.5 ≤ R < 0.75 he/she takes the step in the east direction, and if 0.75 ≤ R ≤ 1.0 he/she takes the step in the west direction. 10.4.2 Poisson Process Often one is interested in the number of events which occur over a certain interval of time, i.e. a counting process (N(t), t ≥ 0). A counting process is called a Poisson process if it involves • One arrival at a time. • Random arrivals without rush or slack periods (stationary increments). • Independent increments. Under these circumstances, the probability that N(t) = n for t ≥ 0 and n = 0, 1, 2, ... is 486 10 Statistical and Probabilistic Models 40 ’RWPath’ 30 20 10 0 -10 -20 -30 -40 -50 -20 -10 0 10 20 Fig. 10.9 The path formed by 10000 steps of a random walker 30 40 50 60 70 10.4 Probabilistic and Statistical Modeling 487 300 ’RandomWalkers’ 200 100 0 -100 -200 -300 -300 -200 -100 0 Fig. 10.10 The final destination after 10000 steps for 5000 random walkers 100 200 300 488 10 Statistical and Probabilistic Models P(N(t) = n) = exp(−λ t)(λ t)n n! This means that N(t) has a Poisson distribution with parameter α = λ t. Its mean and variance are E(N(t)) = V (N(t)) = α = λ t. It can be shown that if the number of arrivals has a Poisson distribution, the inter arrival times have an exponential distribution. The random splitting property of Poisson processes states that if N(t) = N1 (t) + N2 (t) is Poisson with rate λ , then N1 and N2 are independent Poisson with rates λ p and λ (1 − p), where p and1 − p are the probabilities of the branches N1 and N2 . Similarly, if N1 (t) + N2(t) = N(t), the reverse is true (random pooling property). 10.4.3 Markov Chains and the Kolmogorov Balance Equations If the future probability characteristics of a system in which a stochastic process is taking place depend only on the state of the system at the current time, one has a Markov process or chain. The effect of the past on the future is contained in the present state of the system. As a simple example of a Markov chain consider a machine that works until it fails (randomly) and then resumes work once is repaired. There are two states for this system, namely • The machine is busy (S0 ) • The machine is being repaired (S1 ) The system moves from state S0 to S1 at a rate λ and from S1 back to S0 at a rate µ . As a second example consider now a facility where two machines A and B perform an operation. The machines fail randomly but resume work once they are repaired. The four possible states of this system are • • • • Both machines are busy (S0 ) Machine A is being repaired while B is busy (S1 ) Machine B is being repaired while A is busy (S2 ) Both machines are being repaired (S3 ) Now λ1 and λ2 are, respectively, the failure rates of machines A and B while µ1 and µ2 are the corresponding repair rates. The Kolmogorov Balance Equations are differential equations relating the probabilities of the various states involved in a Markov chain P0 , P1, P2 and P3 . They are obtained by a probability balance on the states. For instance, for the second example above they are dP0 = µ1 P1 + µ2 P2 − (λ1 + λ2 )P0 dt 10.4 Probabilistic and Statistical Modeling 489 dP1 = λ1 P0 + µ2 P3 − (µ1 + λ2 )P1 dt dP2 = λ2 P0 + µ1 P3 − (λ1 + µ2 )P2 dt dP3 = λ2 P1 + λ1 P2 − (µ1 + µ2 )P3 dt Under steady state or equilibrium conditions the time derivatives are zero and the probabilities are then related by a system of simultaneous linear algebraic equations. 10.4.4 Queueing Systems and Little’s Formula A queueing system involves one or more servers which provide some service to customers who arrive, line up and wait for service at a queue when all the servers are busy. Typically, both arrival and service times are random variables. The single server queue consists of a single server and a single queue. If the inter arrival times of customers and the service times are exponentially distributed the resulting queue is known as the M/M/1 queue. Inter arrival and service times in queues are often modeled probabilistically. Two examples of queueing systems are: • The arrival of an operator in a machine shop mechanics at a centralized tool crib. • The arrival of a customer at a bank. Random inter arrival and service times are often simulated using exponential distributions. However, sometimes a normal distribution or a truncated normal distribution may be more appropriate. Gamma and Weibull distributions are also used to model arrival and service times. A most important parameter of the queueing system is the server utilization ρ defined as ρ= λ µ where λ is the mean arrival rate of customers from the outside world into the queueing system and µ is the mean service rate. The single server queue can also be regarded as a Markov chain in which the various states are distinguished only by the number of customers waiting in the queue. Let us call the corresponding states S0 , S1, ..., Sn. The system can then move into state Si either from Si−1 (if a new customer arrives before service is completed for the customer being served) or from Si+1 if service is completed and the next 490 10 Statistical and Probabilistic Models customer in line begins service before any new arrival. Let λi, j be the rate at which the system transitions from state Si to state S j . If the queue is at steady state, the Kolmogorov equations yield Pn = λn−1,n ...λ1,2λ0,1 P0 λn,n−1 ...λ2,1λ1,0 where Pn is the probability of encountering n customers in the system and λ0,1 = λ . In investigating queueing systems one is also interested in other measures of system performance such as the expected number of customers in the system (queue plus service station) L, the expected number of customers in the queue Lq , the expected wait time of customers in the system W , and the expected wait time of customers in the queue Wq . The above expectancies are related by Little’s Formula L = λW or that Lq = λ Wq A number of queueing problems have been solved yielding closed form expressions for the above performance parameters. For instance, for the M/M/1 queue at steady state the results are as follows ρ λ µ −λ = 1−ρ 1 1 µ −λ = µ (1−ρ ) 2 ρ2 Lq µ (µλ −λ ) = 1− ρ ρ Wq µ (µλ−λ ) = µ (1− ρ) Pn (1 − λµ )( λµ )n L W Further, for the M/G/1 queue in which the service times have a mean of 1/ µ and a variance σ 2 the corresponding results are ρ 2 (1+σ 2 µ 2) = ρ + 2(1−ρ ) 2(1−ρ ) λ (1/µ 2 +σ 2 ) W µ1 + 2(1−ρ ) 2 2 µ 2 +σ 2 ) σ 2 µ 2) = ρ (1+ Lq λ (1/ 2(1−ρ ) 2(1−ρ ) µ 2 +σ 2 ) Wq λ (1/ 2(1−ρ ) L ρ+λ P0 1 − ρ 2 (1/µ 2 +σ 2 ) 10.4 Probabilistic and Statistical Modeling 491 10.4.5 Matching Field Data with Theoretical Distributions A common situation in reliability analysis and modeling consists of finding probability distribution functions that best fit raw data known to consist of independent and identically distributed (i.i.d.) variables. This is an optimization problem that is solved using standard methods based on the notion of minimizing the squares of the errors. Time to failure of complex components is usually represented with the Weibull distribution. If failures are completely random, the exponential distribution is used. The log-normal distribution can also be used. For incomplete data uniform, triangular and beta distributions are used. As with random numbers, field data must then be tested for independence. Some useful tools are: • Scatter Plots. Contiguous values in a string of values of a random variable are plotted on a x − y plane. The resulting pattern of points is characteristic of the distribution. • Autocorrelation Plots. The covariance of values separated by a specified lag in a string of values of a random variable is plotted as a function of the number of data points. • Runs Tests. This searches for peculiar patterns in substrings of numbers from a larger stream. Data must also be tested to see if they are Identically Distributed (Homogeneity Tests). Some useful tools are: • • • • • • • Histograms. Distribution Plots. Quartile-Quartile Plots. Kolmogorov-Smirnov. Chi-squared. Time Dependency of Distributions. ANOVA. The collected data typically consists of a limited number of data values. Simulation modeling require large numbers of multiple samples therefore the data must be converted to a frequency distribution. Raw data can be used as input for the simulation project but this is usually not recommended except in special cases. More commonly, once data have been tested for independence and correlation they are converted to a form suitable for use in the simulation model. This is done by fitting it to some distribution. Once a distribution fitting the data has been determined, input for the simulation program is produced as random variates sampled from the fitted distribution. The frequency distribution selected can be empirical or theoretical, discrete or continuous. Discrete distributions are rarely directly used. Instead, numerical values of discrete probabilities are directly used. Effectively, continuous, theoretical distributions are almost always employed. 492 10 Statistical and Probabilistic Models Of the many available theoretical distributions 12 or so are commonly used in simulation modeling. Data are fitted to theoretical distributions by identifying the theoretical distribution which best represents the data. Note also that if the fitted distribution is unbounded, values for simulation should be taken rather from a truncated version of the selected distribution in order to avoid unrealistic extreme values. Each statistical distribution function has a physical basis. An understanding of this basis is useful in determining candidate distributions to represent field data. Following is a brief summary of the physical basis of selected distributions. • Binomial. This represents the distribution of a random variable giving the number of successes in n independent trials each yielding either success or failure with probabilities p and 1 − p, respectively. • Geometric. This represents the distribution of a random variable giving the number of independent trials required in an experiment before k successes are achieved. • Poisson. This represents the distribution of a random variable giving the number of independent events occurring within a fixed amount of time. • Normal. This represents the distribution of a random variable which is itself the result of the sum of component processes. • Lognormal. This represents the distribution of a random variable which is itself the result of the product of component processes. • Exponential. This represents the distribution of a random variable giving the time interval between independent events. • Gamma. A distribution of broad applicability restricted to non-negative random variables. • Beta. A distribution of broad applicability restricted to bounded random variables. • Erlang. This represents the distribution of a random variable which is itself the result of the sum of exponential component processes. • Weibull. This represents the distribution of a random variable giving the time to failure of a component. • Uniform. This represents the distribution of a random variable whose values are completely uncertain. • Triangular. This represents the distribution of a random variable for which only minimum, most likely and maximum values are known. 10.4.6 Discrete Event Simulation Modeling Discrete event simulation (DES) models are used to simulate the time evolution of systems with probabilistic inputs and outputs. In DES, the state of the system changes at a discrete set of points in time based on the outcomes of random variate generation processes. Input data for DES models must often be created by inverse transform methods according to a specific statistical distribution. The required distribution must be 10.4 Probabilistic and Statistical Modeling 493 identified based on how well it represents collected data. Following is a brief summary of the real-life situations where the distributions mentioned above are likely to be encountered. • Binomial. Useful when there are only two possible outcomes of an experiment which is repeated multiple times. • Geometric. Useful also when there are only two possible outcomes of an experiment which is repeated multiple times. • Poisson. Useful to represent the number of incoming customers or requests into a system. • Normal. Useful to represent the distribution of errors of all kinds. • Lognormal. Useful for representation of times required to perform a given task or accomplish a certain goal. • Exponential. Useful to represent inter arrival times in all kinds of situations. • Gamma. Useful also for representation of times required to perform a given task or accomplish a certain goal but more general. • Beta. Useful as a rough model under situation of ignorance and/or to represent the proportion of non-conforming items in a set. • Erlang. Useful to represent systems making simultaneous request for attention from a server. • Weibull. Useful to represent the life and/or reliability of components. • Uniform. Useful when one knows nothing about the system. • Triangular. Useful when one know little about the system. As mentioned above, in discrete event simulation modeling the collected input data is often replaced by computer generated random variate values which accurately represent the original data. Typically, several distributions may be considered good candidates for this task. A histogram of the data can provide a first inkling about the family of distribution function(s) which can well represent the data. Another simple test which can be used to quickly determine whether a given set of data are adequately represented by a specific distribution is the construction of quantilequantile plots. Assume that X is a random variable whose cumulative distribution function is F. The q-quantile of X is that value γ of the random variable which satisfies the equation F(γ ) = P(X < γ ) = q j(n+1) In n data values are arranged in increasing order then the value of the k largest value will be denoted by g j/k . Therefore, the median is g1/2 , i.e. the n+1 2 -th or half-way through the data set. A common application of this concept is in investigating the distribution of income in a population where the total number of households is divided into five quintiles, (q = 0.2, 0.4, 0.6, 0.8 and 0.1) by increasing values of income. For instance, in the USA, if one’s household income is more than about γ = 80, 000 dollars per year then one belongs in the top quintile; i.e. one in five households is in that quintile. 494 10 Statistical and Probabilistic Models Consider a collection of n values of the random variable X, xi for i = 1, 2, ..., n. If the values are arranged according to their magnitude a new string of values is obtained which we call y j with j = 1, 2, ..., n. The new variable becomes immediately an estimate for the ( j − 12 )/n quantile of X, i.e. y j ≈ F −1 ( j − 12 ) n Once an appropriate family of distributions has been selected one proceeds to determine the various distribution parameters using the collected data values. To determine the appropriateness of a given distribution in a particular situation goodnessof-fit tests are required. The tests verify the validity of the null hypothesis H0 which states that the random variable X follows a specific distribution. Two commonly used and easily implemented tests are the Chi-square test (applicable to large samples) and the Kolmogorov-Smirnov test (applicable to small samples and restricted to continuous distributions). For the Chi-square test the n data points are arranged into a desired number of cells (k). The expected number of points to fall inside the i-th cell, Ei is then given by Ei = npi where pi is the probability associated with that interval and is obtained from the specified distribution. For instance, consider the case of reliability data consisting of a total of n f failures, binned into cells representing number of failures within time intervals of uniform duration ∆ ti = ti+1 − ti . Assume that direct calculation of the failure rate (hazard function value) yields a reasonably constant trend and that an average value is computed. Introduce then the null hypothesis that the data is exponentially distributed with constant failure rate λ̂ estimated as the average failure rate computed from the data. The expected number of failures inside each time bin is then given by Ei = n f × [exp(−λ̂ ti ) − exp(−λ̂ ti+1 )] Next, using the actual number of data points contained in each cell, Oi one computes the statistic n (Oi − Ei )2 Ei i=1 χ02 = ∑ To test the null hypothesis, the critical value of the statistic is determined as χα2 ,k−s−1 where α is the confidence level and k − s − 1 is the number of degrees of freedom and s is the number of parameters in the candidate distribution (s = 1 in the case of the exponential distribution). Finally, if χ02 > χα2 ,k−s−1 then H0 is rejected but if χ02 < χα2 ,k−s−1 the hypothesis cannot be rejected at the given confidence level. 10.4 Probabilistic and Statistical Modeling 495 If the null hypothesis cannot be rejected one can then calculate a confidence interval for the distribution parameters. For instance, considering again the case of the exponentially distributed reliability data above, one can show that a 100(1 − α )% confidence interval for the value of λ̂ is given by [λ̂ × χ02 (2n f , 1 − α /2) χ 2 (2n f , α /2) , λ̂ × 0 ] 2n f 2n f For the Kolmogorov-Smirnov test the n data points are also arranged in increasing order. If possible the data are made dimensionless dividing each value by the largest value in the set. Then, one calculates the statistics i D+ = max( − Ri ) n D− = max(Ri − i−1 ) n and D = max(D+ , D−) and compares D against the critical value Dc . When D < Dc the null hypothesis H0 cannot be rejected. Sometimes input data for DES models is just not easily available. In those instances one must rely on related engineering data, expert opinion and physical or other limitations to produce reasonable input values for the model. A few data values combined with the assumption of uniform, triangular or beta distribution can provide a solid starting point for research. In other situations, various inputs may be related to each other or the same input quantity may exhibit autocorrelation over time and this fact needs to be taken into account for proper simulation. Typical examples are in inventory modeling where demand data affect lead time data and in stock trading where buy and sell orders called to the broker tend to arrive in bursts. When two correlated input variables X1 and X2 are involved one uses their covariance Cov(X1 , X2) or their correlation ρ= Cov(X1 , X2 ) σ2 σ1 If collected data values for the two values yields ρ >> 0 then one needs to generate correlated variates. The following algorithm generates two correlated random variates with normal distributions with parameters µ1 , σ1 and µ2 , σ2 , respectively. • Generate two independent standard normal variates Z1 and Z2 . • Set X1 = µ1 + σ1 Z1 496 10 Statistical and Probabilistic Models • Set X2 = µ2 + σ2 (ρ Z1 + p 1 − ρ 2 Z2 ) 10.4.7 Reliability Modeling Reliability analysis is the investigation of the probability that a given system may fail to perform as expected. In reliability modeling probabilistic simulation methods are used to determine estimates of the useful lives of systems. In reliability, the random variable of interest is the time to failure of an item T . This time to failure may be calendar time or any other number indicative of the life of the system. The reliability function R(t) of an item is the probability that the failure time of an item will exceed time t, i.e. R(t) = Pr(T > t) = 1 − F(t) where F(t) is the cumulative probability distribution function of the failure times, i.e. F(t) = Pr(T ≤ t) = Z t f (τ )d τ 0 representing the probability that the failure time of the item will be less that t. Here, f (t) is the corresponding probability density function of failure time. In terms of the density function, the reliability function is R(t) = 1 − Z t f (τ )d τ = 0 Z ∞ f (τ )d τ t A very important quantity, the mean time to failure T̄ f of an item is then calculated as T̄ f = Z ∞ R(t)dt 0 The conditional probability that the item will fail in the time interval between t and t + ∆ t given that it is still functioning at time t is given by Pr(t < T ≤ t + ∆ t|T > t) = Pr(t < T ≤ t + ∆ t) F(t + ∆ t) − F(t) = Pr(T > t) R(t) The failure rate or force of mortality or hazard rate function z(t) is a very important quantity since it measures the probability that an item that has survived till time t will fail some time between t and t + ∆ t. Therefore, z(t) is given as z(t) = lim ∆ tto0 f (t) Pr(t ≤ T ≤ t + ∆ t|T > t) 1 F(t + ∆ t) − F(t) = lim = ∆t ∆ tto0 R(t) ∆t R(t) 10.4 Probabilistic and Statistical Modeling 497 The four functions introduced above are all interrelated and knowledge of one of them determines the other three. The reliability of systems formed by various components is a function of the individual reliabilities of said components. A binary variable xi can be used to describe whether a component i is working (1) or not (0). The structure function of a system formed by multiple components is a simple binary variable that takes the value of 1 when the system is functioning and zero when it fails. Components in systems may be arranged in series mode or in parallel mode. The structure function φ of a system of n components arranged in series mode is simply defined as n φ (xi ) = x1 x2 ...xn = ∏ xi i=1 and that of the system where the components are arranged in parallel by n φ (xi ) = 1 − (1 − x1 )(1 − x2 )...(1 − xn) = 1 − ∏(1 − xi ) i=1 The corresponding system reliabilities are, for the system of n components in series n Rs = R1 R2 ...Rn = ∏ Ri i=1 and n n R p = 1 − (1 − R1 )(1 − R2 )...(1 − Rn) = 1 − ∏(1 − Ri ) = 1 − ∏(Fi ) i=1 i=1 for the one with the components arranged in parallel. Here Ri , i = 1, 2, ..., n are the reliabilities of the individual components. More complex systems can be represented as composite arrangements of subsystems some arranged in series y some in parallel. Therefore, a system with n components that still functions when at least k of components are still functioning can be regarded as a collection of n series sub-systems arranged in parallel, each containing k components. This arrangement is called a k-out-of-n structure As an example consider an aircraft containing three engines but designed so that it can continue flying even when one of the engines fails. When any of the engines fails, the other two still work and the plane can keep flying. As an example consider a system consisting of three components, 1,2,3 with Weibull failure probability distribution functions given respectively as follows: F1 = 1 − exp(−( t 0.5 ) ) 10 498 10 Statistical and Probabilistic Models F2 = 1 − exp(− F3 = 1 − exp(−( t ) 10 t 5 ) ) 10 If the components are arranged in series Rs = R1 R2 R3 = (1 − F1 )(1 − F2 )(1 − F3 ) = = exp(−0.3162t 0.5) exp(−0.1t) exp(−10−5t 5 ) The mean time to failure is calculated as T̄s = 3.4. If one assumes instead that the components are connected in parallel = 1 − (1 − exp(−0.3162t 0.5 R p = 1 − (1 − R1 )(1 − R2 )(1 − R3 ) = ))(1 − exp(−0.1t))(1 − exp(−10−5t 5 )) The mean time to failure is T¯p = 27.19. Finally for the 2-out-of-3 arrangement, the reliability is R2oo3 = R1 R2 + R2 R3 + R3 R1 − 2R1 R2 R3 The value of T¯p = 8.58. The figure below shows the values of the hazard function z(t) for the three cases considered. For the series arrangement, the curve exhibits the often encountered bathtub shape associated with an initial burn-in period, followed by a useful life stage and then a wear out period. The other two curves are somewhat similar to each other but the parallel arrangement exhibits the lowest values of z(t) while the 2-out-of-3 arrangement yields intermediate values but the shape of the curve resembles more the one corresponding to the parallel arrangement. Very similar results are obtained for this problem using the Monte Carlo method of simulating the failures using random variates drawn from the given probability distributions and sufficiently large samples. As another example consider the case of deciding when best to replace an item if its failure probability distribution function is known. The mean time between replacements with replacement age t0 , T̄ )r is T̄r = Z t0 0 t f (t) + t0Pr(T ≥ t0 ) = Z t0 0 (1 − F(t))dt Note that if replacement only takes place when the item fails, the mean time between replacements is the mean time to failure T̄ f . Let F(t) be given by F = 1 − exp(−( t 3 ) ) 10 10.4 Probabilistic and Statistical Modeling 499 Fig. 10.11 The hazard function z(t) for three different arrangements of the three component system discussed. 500 10 Statistical and Probabilistic Models If the item is replaced before it fails, the cost incurred is c = 5. On the other hand, if one waits until the item fails, one must incur additional (liability) expenses such that the costs of replacement (following failure) is k = rc where r > 1 depending of the severity of the consequences of failure. The objective is to determine an optimal replacement age t0 for the component. The mean total cost per replacement period c̄ is c̄ = c + kPr(T, t0 ) = c + kF (t0 ) Alternatively, the total cost per replacement period if replacement takes place only when the item fails is c̄∞ = c + k The mean total cost per unit time with replacement age t0 , C(t0) is given by C̄ = c̄ T̄r If the item is replaced only when it fails, the mean cost per unit time is instead given by C̄∞ = c̄∞ T̄ f This quantity cannot be expressed in terms of elementary functions for the given F(t) but it can be determined by symbolic manipulation or Monte Carlo simulation. The figure below shows the graph of C̄ as a function of the replacement time t0 and the coefficient r. The picture shows that for every value of r there is an optimal replacement time t0 . The optimal replacement time t0∗ is determined from the extremum condition dC̄ =0 dt0 For instance, for r = 10, the resulting value of t0∗ = 3.69 and this yields C̄ = 2.04. This should be compared against the corresponding cost resulting from replacing only at failure which is C̄∞ = 6.16. Hence replacement at age t0∗ = 3.69 is cheaper than waiting until the item fails. When the liability costs are higher (say r = 100), the optimal replacement time decreases to t0∗ = 1.71. However, the associated costs increases to C̄ = 4.38 since the item must be replaced more frequently. Clearly, it may well happen that it is more costly to replace before failure despite the additional cost of liability. 10.4 Probabilistic and Statistical Modeling 501 Fig. 10.12 The function C̄(t0 , r) representing the mean replacement cost per unit time for the case discussed. 502 10 Statistical and Probabilistic Models 10.5 Exercises 1.- Use the linear congruential method to produce 10000 pseudo random numbers uniformly distributed between 0 and 1. Examine the results. 2.- Use the inverse transform method to produce random variate values for random variables from the following distributions: • • • • • • • Uniform U(0, 6) Triangular T (0, 3, 6) Normal N(3, 1.5) Exponential E(0.5) Weibull W (3.5, 2) Log-Normal LN(3, 1.5) Gumbel G(0, 6) Examine and compare the results. 3.- Obtain an actuarial life table and use the data in it to identify a probability distribution that fits the data. 4.- A machine works continuously until it randomly fails. The machine is immediately sent for repair and resumes work once is repaired. Make a graph representing the above Markov chain. 5.- At a facility, two machines perform an operation. The machines fail at random but resume work once they are repaired. If the failure and repair rates of the machines are λ1 and λ2 and µ1 and µ2 , identify the possible states of this system are and make a Markov chain graph of them. 6.- Derive the Kolmogorov equations for the single server queue. 7.- Derive Little’s formula relating L and W . 8.- Estimate the probability that a male in the US that has survived till his sixtieth birthday will pass away the day after his birthday. Then create a Monte Carlo simulation model to estimate the probability. 9.- Consider a system consisting of three components, 1,2,3 with Weibull failure probability distribution functions given respectively as follows: F1 = 1 − exp(−( t 0.5 ) ) 10 10.6 References 503 F2 = 1 − exp(− F3 = 1 − exp(−( t ) 10 t 5 ) ) 10 Assume the components can be arranged in series, in parallel or in a 2-out-of3 arrangement. Create Monte Carlo simulation models for the three different arrangement of this system and use them to estimate the corresponding hazard rate functions. 10.- Consider the problem of determining the optimum replacement time t0∗ for an item whose failure probability distribution function is given by F = 1 − exp(−( t 3 ) ) 10 If the item is replaced before it fails, the cost incurred is c = 5. On the other hand, if one waits until the item fails, one must incur additional (liability) expenses such that the costs of replacement (following failure) is k = 100c. Create a Monte Carlo simulation model for this system and use it to estimate the optimal replacement time. 10.6 References 1.- Banks, J. et al., Discrete-Event System Simulation, 3rd ed., Prentice-Hall, Englewood Cliffs, NJ, 2001. 2.- Cyganowski, S. et al., From Elementary Probability to Stochastic Differential Equations with Maple, Springer, Berlin, 2002. 3.- Gnedenko, B.V. and Khinchin, A.Ya., An Elementary Introduction to the Theory of Probability, Dover, NY, 1962. 4.- Grimmett, G.R. and Stirzaker, D.R., Probability and Random Processes, 2nd. ed., Clarendon Press, Oxford, 1995. 5.- Johnson, R.A. et al., Miller and Freund’s Probability and Statistics for Engineers, 8th ed., Pearson, Harlow, UK, 2014. 6.- Hogg, R.V. et al., Introduction to Mathematical Statistics, 6th ed., Pearson/PrenticeHall, Upper Saddle River, NJ, 2005. 504 10 Statistical and Probabilistic Models 7.- Kreyszig, E., Advanced Engineering Mathematics, 9th. ed., Wiley, NY, 2006. 8.- Rausand, M. and Hoyland, A., System Reliability Theory: Models, Statistical Methods and Applications, Wiley, Hoboken, 2004. 9.- Ross, S.M., Introduction to Probability Models, 11th ed. Academic Press/Elsevier, Amsterdam, 2014. 10.- Rubinstein, R.Y. and Kroese, D.P., Simulation and the Monte Carlo Method, Wiley, Hoboken, 2008. 11.- Rychlik, I. and Ryden, J., Probability and Risk Analysis: An Introduction for Engineers, Springer, Berlin, 2006.