Download N - MIND

Analysis of Simulated
Data
Media
 
Media di una popolazione: somma di tutti i valori delle
variabili della popolazione diviso il numero di unità della
popolazione (N)
N
µ=
 
∑X
Dove:
- N = numero elementi popolazione
- Xi =i-esima osservazione della variabile Xi
i
i =1
N
Media di un campione: somma di tutti i valori delle variabili
di un sottoinsieme della popolazione diviso il numero di
unità di tale campione (n)
n
X =
∑X
i =1
n
i
Varianza
 
Varianza della popolazione: misura che caratterizza molto
bene la varibilità di una popolazione
N
σ2 =
 
2
(
)
X
−
µ
∑ i
i =1
N
Varianza di un campione:
n
s2 =
 
Dove:
- N è il numero di osservazioni dell’intera popolazione
- µ è la media della popolazione
- xi è l’i-esimo dato statistico osservato
∑ (X
i
−X
i =1
n −1
2
)
Dove:
- n è il numero di osservazioni del campione
- X è la media del campione
- xi è l’i-esimo dato statistico osservato
Quando n è grande le differenze tra le due formule sono minime;
quando n è piccolo, le differenze sono sensibili.
Teorema Centrale Limite
Quando la
numerosità del
campione diventa
abbastanza grande
La distribuzione delle
medie campionarie
approssima una
normale
X
X
Quando la popolazione non segue una normale
Central Tendency
µx = µ
Population Distribution
σ = 10
Variation
σ
σx =
n
µ = 50
X
Sampling Distributions
µ X = 50
X
Distribuzione campionaria
Suppose there’s a
population...
 
 
 
 
B
C
Random variable, X,
is Age of individuals
Values of X: 18, 20, 22, 24
measured in years
EVERYONE is one of these 4 ages in
this population
D
A
© 1984-1994 T/Maker Co.
Caratteristiche della popolazione
N
µ=
∑X
Population Distribution
i
i =1
N
P(X)
18 + 20 + 22 + 24
=
= 21
4
.3
.2
.1
0
N
σ=
∑ (X
2
i
i =1
N
− µ)
= 2.236
A
B
C
D
(18)
(20)
(22)
(24)
Uniform Distribution
X
Possibili campioni di dim. n = 2
st
1
Obs
nd
2
18
Observation
20
22
24
18 18,18 18,20 18,22 18,24
20 20,18 20,20 20,22 20,24
16 Sample Means
22 22,18 22,20 22,22 22,24
1st 2nd Observation
Obs 18 20 22 24
24 24,18 24,20 24,22 24,24
18 18 19 20 21
16 Samples
Samples Taken with
Replacement
20 19 20 21 22
22 20 21 22 23
24 21 22 23 24
Distribuzione campionaria
(di tutte le medie campionarie)
16 Medie campionarie
Distribuzione delle
medie campionarie
1st 2nd Observation
Obs 18 20 22 24
P(X)
18 18 19 20 21
.3
20 19 20 21 22
.2
22 20 21 22 23
.1
24 21 22 23 24
0
# nel campione = 2,
_
18 19
20 21 22 23
24
X
# nella distrib. campionaria = 16
Media e deviazione standard della
distrib. campionaria
N
µx =
∑X
i =1
N
N
σx =
=
∑ (X
i =1
i
18 + 19 + 19 +  + 24
=
= 21
16
2
i
− µx )
N
(18 − 21)2 + (19 − 21)2 +  + (24 − 21)2
16
= 1.58
Confronto popolazione/distrib. campionaria
Popolazione
Distr. Medie campionarie
n=2
µ = 21, σ = 2.236
P(X)
.3
P(X)
.3
.2
.2
.1
.1
0
0
A
B
C
(18)
(20)
(22)
D X
(24)
µ x = 21
18 19
σ x = 1.58
20 21 22 23
24
_
X
Curva Normale: proprietà
Valore approssimato della percentuale dell’area compresa tra
valori di deviazione standard (regola empirica).
99.7%
95%
68%
Confidence Interval for a Mean
when you have a “small”
sample...
As long as you have a “large” sample….
A confidence interval for a population mean is:
 s 
x ± Z

 n
where the average, standard deviation, and n depend
on the sample, and Z depends on the confidence level.
Example
Random sample of 59 students spent an
average of $273.20 on Spring 1998
textbooks. Sample standard deviation
was $94.40.
 94.4 
273.20 ± 1.96
 = 273.20 ± 24.09
 59 
We can be 95% confident that the average amount spent
by all students was between $249.11 and $297.29.
What happens if you can only take a
“small” sample?
  Random
sample of 15 students slept an
average of 6.4 hours last night with
standard deviation of 1 hour.
  What is the average amount all students
slept last night?
If you have a “small” sample...
Replace the Z value with a t value to get:
 s 
x ± t

 n
where “t” comes from Student’s t distribution,
and depends on the sample size through the
degrees of freedom “n-1”.
Student’s t distribution versus
Normal Z distribution
T-distribution and Standard Normal Z distribution
0.4
Z distribution
density
0.3
0.2
T with 5 d.f.
0.1
0.0
-5
0
Value
5
T distribution
 
Very similar to standard normal distribution,
except:
 
t depends on the degrees of freedom “n-1”
 
more likely to get extreme t values than extreme
Z values
Let’s compare t and Z values
Confidence t value with Z value
level
5 d.f
90%
2.015
1.65
95%
2.571
1.96
99%
4.032
2.58
For small samples, T value is larger than Z value.
So, T interval is made to be longer than Z interval.
OK, enough theorizing!
Let’s get back to our example!
Sample of 15 students slept an average of 6.4
hours last night with standard deviation of 1 hour.
Need t with n-1 = 15-1 = 14 d.f.
For 95% confidence, t14 = 2.145
 s 
 1 
x ± t
 = 6.4 ± 2.145
 = 6.4 ± 0.55
 n
 15 
That is...
We can be 95% confident that average amount
slept last night by all students is between 5.85
and 6.95 hours.
What happens as
sample gets larger?
T-distribution and Standard Normal Z distribution
0.4
Z distribution
density
0.3
T with 60 d.f.
0.2
0.1
0.0
-5
0
Value
5
What happens to CI as
sample gets larger?

x ± Z


x ± t

s 

n
s 

n
For large samples:
Z and t values
become almost
identical, so CIs are
almost identical.
Example
Random sample of 64 students spent an average of 3.8
hours on homework last night with a sample standard
deviation of 3.1 hours.
Z Confidence Intervals The assumed sigma = 3.10
Variable
Homework
N
Mean
64 3.797
StDev
3.100
95.0 % CI
(3.037,
4.556)
T Confidence Intervals
Variable N
Mean
Homework 64 3.797
StDev
3.100
95.0 % CI
(3.022,
4.571)
Output analysis for single system
Why?
 
 
 
 
 
Often most of emphasis is on simulation model development
and programming.
Very little resources (time and money) is budgeted for
analyzing the output of the simulation experiment.
In fact, it is not uncommon to see a single run of the
simulation experiment being carried out and getting the
“results” from the simulation model.
The single run also is of arbitrary length and the output of this
is considered “true.”
Since simulation modeling is done using random parameters of
different probability distributions, this single output is just one
realization of these random variables.
Why?
 
If the random parameters of the experiment may have a large
variance, one realization of the run may differ greatly from the
other.
 
This is a real danger of making erroneous inferences about the
system we are trying to simulate because we know that
a single data point has practically no statistical
significance !!!
Why?
 
A simulation experiment is a computer-based statistical
sampling experiment, hence if the results of the simulation are to
have any significance and the inferences to have any confidence,
appropriate statistical techniques must be used !!
 
Most of the times output data of the simulation experiment is
non-stationary and auto-correlated. Hence classical statistical
techniques which require data to be IID can’t be directly
applied.
Typical output process
 
 
Let Y1, Y2, … Ym be the output stochastic process from a single simulation
run.
Let the realizations of these random variables over n replications be:
y11 y12  y1m
y21 y22  y2 m

yn1 yn 2  ynm
 
 
It is very common to observe that within the same run the output process
is correlated. However, independence across the replications can be
achieved.
The output analyses depends on this independence.
Transient and steady-state behavior
 
Consider the stochastic processes Yi as before.
 
In many experiment, the distribution of the output process
depends on the initial conditions to certain extent.
 
This conditional distribution of the output stochastic process
given the initial condition is called the transient distribution.
 
If this sequence converges, as i → ∞ for any initial condition,
then we call the convergence distribution as steady-state
distribution.
Types of simulation
 
Terminating simulation
 
Non-terminating simulation
Steady-state parameters
Steady-state cycle parameters
Others parameters
o 
o 
o 
Terminating simulation
 
 
 
When there is a “natural” event E that specifies the length of each run
(replication).
If we use different set of independent random variables at input, and same
input conditions then the comparable output parameters are IID.
Often the initial conditions of the terminating simulation affect the
output parameters to a great extent.
Examples of terminating simulation:
1.  Banking queue example – when specified that bank operates
between 9 am to 5 pm.
2.  Inventory planning example (calculating cost over a finite time
horizon).
Non-terminating simulation
 
There is no natural event E to specify the end of the run.
 
Measure of performance for such simulations is said to be steady-state
parameter if it is a characteristic of the steady-state distribution of some
output process.
 
Stochastic processes of most of the real systems do not have steady-state
distributions, since the characteristics of the system change over time.
 
On the other hand, a simulation model may have steady-state distribution,
since often we assume that characteristics of the model don’t change with
time.
Non-terminating simulation
 
Consider a stochastic process Y1, Y2, … for a non-terminating simulation
that does not have a steady-state distribution.
 
Now lets divide the time-axis into equal-length, contiguous time intervals
called cycles. Let YiC be the random variable defined over the ith cycle.
 
Suppose this new stochastic process has a steady-state distribution.
 
A measure of performance is called a steady-state performance it is
characteristic of YC.
Non-terminating simulation
 
For a non-terminating simulation, suppose that a stochastic process does
not have a steady-state distribution.
 
Also suppose that there is no appropriate cycle definition such that the
corresponding process has a steady-state distribution.
 
This can occur if the parameters for the model continue to change over
time.
 
In these cases, however, there will typically be a fixed amount of data
describing how input parameters change over time.
 
This provides, in effect, a terminating event E for the simulation, and, thus,
the analysis techniques for terminating simulation are appropriate.
Statistical analysis of terminating simulation
 
Suppose that we have n replications of terminating simulation, where each
replication is terminated by the same event E and is begun by the same
“initial” conditions.
 
Assume that there is only one measure of performance.
 
Let Xj be the value of performance measure in jth replication j = 1, 2, …n.
So these are IID variables.
N
 
For a bank, Xj might be the average waiting time
W
∑
(
i
i =1
) over a
N
day from the jth replication where N is the number of customers served in
a day. We can also see that N itself could be a random variable for a
replication.
Statistical analysis of terminating simulation
 
For a simulation of war game Xj might be the number of tanks destroyed
on the jth replication.
 
Finally for a inventory system Xj could be the average cost from the jth
replication.
 
Suppose that we would like to obtain a point estimate and confidence
interval for the mean E[X], where X is the random variable defined on a
replication as described above.
 
Then make n independent replications of simulation and let Xj be the
resulting IID variable in jth replication j = 1, 2, …n.
Statistical analysis of terminating simulation
 
We know that an approximate 100(1- α) confidence interval for µ = E[X]
is given by:
X n ± t n −1,1−α / 2
S 2 ( n)
.
n
where we use a fixed sample of n replications and take the sample variance
from this (S2(n)).
  Hence this procedure is called a fixed-sample-size procedure.
Statistical analysis of terminating simulation
 
 
 
 
One disadvantage of fixed-sample-size procedure based on n
replications is that the analyst has no control over the confidence
interval half-length (the precision of (X n )).
If the estimateX n is such that X n − µ = β then we say that X n has an
absolute error of β.
Suppose that we have constructed a confidence interval for µ based on
fixed number of replications n.
We assume that our estimate of S2(n) of the population variance will
not change appreciably as the number of replications increase.
Statistical analysis of terminating simulation
 
Then, an expression for the approximate total number of replications
required to obtain an absolute error of β is given by:
2


S
(i )

*
na (β ) = min i ≥ n : ti −1,1−α / 2
≤ β .
i


 
If this value na*(β) > n, then we take additional replications (na*(β) – n) of
the simulation, then the estimate mean E[X] based on all the replications
should have an absolute error of approximately β.
Statistical analysis of terminating simulation
Sequential procedure for estimating the confidence interval for .
  Let
δ (k , α ) = t k −1,1−α / 2
S 2 (k )
.
k
1. 
Make k0 replications of the simulation and set k = k0.
2. 
Compute X n and δ(k, α) from the current sample.
3. 
If δ(k, α) < β then use this
4. 
Otherwise replace k with k + 1, make an additional replication of the
simulation and go to Step 1.
X nas a point estimate of
and stop.
A method for determining when to stop
 
 
 
 
Choose an acceptable value d for the standard deviation of the estimator
Generate at least 100 data values
Continue to generate additional data values, stopping when you have
generated k values and
where S is the sample standard deviation based on k values
The estimate of
is given by
(come riportato nel libro di testo)
Example
 
 
Consider a serving system in which no new customer are allowed to
enter after 5 p.m. and we are interested in estimating the expected time at
which the last customer departs the system.
Suppose we want to be at least 95% certain that our estimated answer
will not differ from the true value by more than 15 seconds
Choosing initial conditions
 
The measures of performances for a terminating simulation depend
explicitly on the state of system at time 0.
 
Hence it is extremely important to choose initial condition with utmost
care.
 
Suppose that we want to analyze the average delay for customers who
arrive and complete their delays between 12 noon and 1 pm (the busiest for
any bank).
 
Since the bank would probably be very congested by noon, starting the
simulation then with no customers present (usual initial condition for any
queuing problem) is not be useful.
 
We discuss two heuristic methods for this problem.
Choosing initial conditions
First approach
  Let us assume that the bank opens at 9 am with no customers present.
  Then we start the simulation at 9 am with no customers present and run it
for 4 simulated hours.
  In estimating the desired expected average delay, we use only those
customers who arrive and complete their delays between noon and 1 pm.
  The evolution of the simulation between 9 am to noon (the “warm-up
period”) determines the appropriate conditions for the simulation at noon.
Disadvantage
  The main disadvantage with this approach is that 3 hours of simulated time
are not used directly in estimation.
  One might propose a compromise and start the simulation at some other
time, say 11 am with no customers present.
  However, there is no guarantee that the conditions in the simulation at noon
will be representative of the actual conditions in the bank at noon.
Choosing initial conditions
Second approach
  Collect data on the number of customers present in the bank at noon for
several different days.
 
Let pi be the proportion of these days that i customers (i = 0, 1, …) are
present at noon.
 
Then we simulate the bank from noon to 1 pm with number of customers
present at noon being randomly chosen from the distribution {pi}.
 
If more than one simulation run is required, then a different sample of {pi}
is drawn for each run. So that the performance measure is IID.
Calcolo delle probabilità usando i dati
#Clienti alle 12
0
1
2
3
4
#di giorni
80
50
40
10
20
200
x
0
1
2
3
4
f (x )
.40
.25
.20
.05
.10
1.00
Statistical analysis of steady-state parameters
 
Let Y1, Y2, … Ym be the output stochastic process from a single run of a
non-terminating simulation.
 
Suppose that P(Yi <= y) = Fi(y) → F(y) = P(Y <= y) as i goes to ∞.
 
Here Y is the steady state random variable of interest with distribution
function F.
Then φ is a steady-state parameter if it is a characteristic of Y such as E[Y],
F(Y).
 
 
One problem in estimating φ is that the distribution function of Yi is
different from F, since it is generally not possible to choose i to be
representative of the “steady state behavior.”
Statistical analysis of steady-state parameters
 
This causes an estimator based on observations Y1, Y2, … Ym not to be
“representative.”
 
This is called the problem of initial transient.
 
Suppose that we want to estimate the steady-state mean E[Y], which is
generally given as:
υ = lim E[Yi ].
i →∞
 
Most serious problem is:
E[Ym ] ≠ υ for any m.
Statistical analysis of steady-state parameters
 
 
The technique that is most commonly used is the warming up of the
model or initial data deletion.
The idea is to delete some number of observations from the beginning of
a run and to use only the remaining observations to estimate the mean.
So:
m
∑Y
i
Y (m, l ) =
 
 
i =l +1
m−l
.
Question now is: How to choose the warm-up period l?
We can find the point in which the transient mean curve E[Yi] “flattens
out” at level ν.
Statistical analysis of steady-state parameters
Bootstrapping the Mean: An example
 
 
 
We are interested in finding the confidence interval for a mean
from a sample of only 4 observations.
Assume that we are interested in the difference in income
between husbands and wives: we have four cases, with the
following income differences (in $1000s): 6, -3, 5, 3, for a mean
of 2.75, and standard deviation of 4.031
We can calculate the confidence interval:
S 2 ( n)
µ = X n ± t n −1,.025
n
4.031
= 2.75 ± 4.30 ×
4
= 2.75 ± 4.30 × 2.015 = 2.75 ± 8.66
 
Now we’ll compare this confidence interval to one found using
bootstrapping
Defining the Random Variable
 
 
 
The first thing that bootstrapping does is
estimate the population distribution of X from
the four observations in the sample
In other words, the random variable X* is
defined:
x* p (x* )
6
.25
-3
.25
5
.25
3
.25
The mean of X* is then simply the mean of the
sample X = E ( X *) =2,75
The Sample as the Population
We now treat the sample as if it were the population, and
resample from it
  In this case we take all possible samples with
replacement, meaning that we take nn=44=256 different
samples
  Since we found all possible samples, the mean of these
samples is simply the original mean
  The standard error of X from these samples is:
nn
SE * ( X *) =
 
*
2
(
X
−
X
)
∑ k
k =1
n
n
= 1.745
We now make an adjustment for the sample size
n
SE ( X ) =
SE * ( X *) = 2.015
n −1
The Sample as the Population
 
 
 
 
 
 
 
In this example, because we used all possible resamples
of our sample, the bootstrap standard error (2.015) is
exactly the same as the original standard error
Still, the same approach can be used for statistics for which
we do not have standard error formulas, or we have small
sample sizes
In summary, the following analogies can be made to
sampling from the population:
– Bootstrap observations → original observations
– Bootstrap mean → original sample mean
– Original sample mean → unknown population mean µ
– Distribution of the bootstrap means → unknown
sampling distribution from the original sample