Download s10.pdf

Session 10
Comparing Alternative Designs
Ernesto Gutierrez-Miravete
Spring 2002
1
Comparing Two Designs
One important application of DES modeling is as an aid in discriminating among alternative
designs. Assume we are interested in the value of a single measure of system performance
with mean value of . Since we may have various alternative designs which we want to
compare the corresponding mean performance parameter values are i with i = 1; 2; :::; N
where N is the total number of alternative designs to be compared. We shall focus here on
the comparison between two designs (i.e. N = 2). A DES model is then used to investigate
the dierence . Specically we want to determine point and interval estimates for this
dierence.
To start, the following simulation parameters must be dened for each desing i
Run length, TEi ,
Number of replications, Ri,
Estimate or expected value of the mean performance measure for replication r, Yr;i,
Average and standard deviation of the estimates over replications, Y:i and Si , respectively.
The location of the condence interval with respect to the origin determine if there is
statistically signicant evidence that the designs have dierent performance. A two-sided
100(1 )% condence interval will always have the following form
(Y: Y: ) t= ; s:e:(Y: Y: )
1
2
2
1
2
1
2
2
Here t= ; is the 100(1 =2) percentage point of a t-distribution with degrees of freedom
and s:e:() stands for the standard error of the estimator.
2
1
The expected or average value of the performance parameter over all replications for
design i is
R
Y = 1 X Y
i
:;i
Ri
ri
1
The standard error, on the other hand, depends on the relationship between the variances
of the sample means. Specically, if the simulation experiments for the two designs are
conducted using dierent random number streams one obtains statistically independent
sampling and if it is reasonable to assume that the variances are approximately equal
(i.e. = = ), the standard error is given by
1
2
s:e:(Y:1
Y:2 ) = Sp
s1
1
+
R R
1
2
where Sp is a pooled estimate of given by
(R 1)S + (R 1)S
Sp =
R +R 2
with
R
1 X
Si =
(Yr;i Y:;i )
Ri 1
for i = 1; 2. Finally, the number of degrees of freedom is
=R +R 2
If the samples are still statistically independent but their variances are unequal,
the standard error is given instead by
2
1
1
2
2
2
1
2
i
2
2
1
1
s:e:(Y:1
2
Y:2 ) =
s
S12 S22
+
R1 R2
and the number of degrees of freedom is
(S =R + S =R )
=
[(S =R ) =(R 1)] + [(S =R ) =(R 1)]
Correlated sampling can be obtained if the same random number stream is used to
simulate both systems. This usually results in a reduction in the variance of Y: Y: according
to
var(Y: Y: ) = var(Y: ) + var(Y: ) 2 cov(Y: ; Y: )
2
1
2
1
1
2
2
2
1
2
2
2
1
2
2
2
2
1
1
2
1
2
2
1
2
2
This makes the estimator is more precise. Dene now the dierences in the value of the
estimator for each replication Dr = Yr Yr , so that the sample mean dierence is
1
2
R
D = 1 X Dr
R
1
where R = R = R . With the above, the required standard error is then
1
2
s:e:(Y:1
Y:2 ) = s:e:(D ) =
pSD
R
where the sample variance of the dierences SD is
2
R
1 X
(Dr D )
R 1
Exercise. Consider a three-stage inspection system in which entities arrive with exponential interarrival times of mean = 6.3 minutes. The inspection times for all three stages
follow normal distributions with means of 6.5, 6 and 5.5 minutes, respectively and equal
standard deviation of 0.5 minutes. Two designs of the inspection system are proposed: a
parallel arrangement where each of three servers performs the three inspection steps and
a serial arrangement where each inspection step is performed by a dedicated server. It is
anticipated that the increased specialization resulting from the serial arrangement will allow
reduccions in the mean inspection times per step to new values of 5.85, 5.4 and 4.95 minutes,
respectively. Compare the performance of the two designs for 16 hours of operation using 10
replications. Investigate the eect of using independent sampling, correlated sampling and
correlated synchronized sampling in the model on the computed condence intervals.
SD2 =
2
2
1
Comparing Several Designs
Sometimes several designs must be compared. Let the performance measure of interest for
design i be i for i = 1; 2; :::; K . Comaprisons can be made using xed sample size procedures or sequential sampling procedures. Several possible objectives of a simulation
study in this case include
Estimation of each i,
Comparison of i with a control ,
Investigation of all possible comparisons,
Selection of the best i.
1
3
The estimation of single i's and their variability requires determination of C = K
separate 100(1 i ) condence intervals. The i 's may refer either to a single measure of
performance for multiple designs or to multiple measures of performance for a single design.
In the rst case and for independent sampling, the overall condence level is
C (1 i )
While in the second case the sampling is necessarily correlated and the overall condence
level is instead
1
C
X
1
i = 1
E
1
Here, the overall error probability is dened as
E =
XC 1
i
The signicance of E is that it provides an upper bound to the probability that one or more
of the condence intervals does not contain the parameter being estimated. This is called
Bonferoni inequality.
If K 1 designs are to be compared against a baseline system 1 then C = K 1
separate 100(1 i ) condence intervals for the dierences i just as when comparing
two alternatives.
When performing all possible comparisons among K designs one needs to compute C =
K (K 1)=2 condence intervals for PP
i j , (100(1 ij ). And the bound on the overall
condence coeÆcient is 1 E = 1
i6 j ij .
Exercise. For the same system of the previos exercise consider the following two additional alternative designs.
Series arrangement but with a buer of capacity 1 added between the rst and second
inspection steps.
Series arrangement but with a buer of capacity 1 added between the second and third
inspection steps.
Compare the three series arrangements against the parallel arrangment.
If the best among a set of K alternative designs is to be determined one must focus on
the quantity i maxj6 ij for i = 1; 2; :::; K . One wants the probability of selecting the
best system to be at least 1 after specifying the value of a practically signicant
performance dierence . The following two-stage Bonferoni procedure is commonly
used to determine the system with the largest value of the performance measure.
Specify , the value of 1 and an initial stage sample size R 10. Make t =
t= K ;R0 .
1
=
=
0
(
1)
1
4
3
Make R replications for each design to obtain Yr;i.
Compute mean values (over the replications) for each design.
Calculate the sample variance of the dierences Sij and select the largest.
Calculate a second stage sample size R.
Perform R R additional replications.
Calculate the means over the new replications.
Select the system with the largest mean value.
0
2
0
Statistical Models
Design of experiments techniques can also be used to evaluate multiple alternatives. In single
factor experiments there is a single decision factor D for the response variable Y and
one is usually interested in the eect of level j of the factor D, j on the value of Y . The
simplest statistical model for the experiment is represented by the expresion
Yrj = + j + rj
Here Yrj is the r-th observation for level j , is the mean overall eect and ij is the random
variation in observation r for level j . One can perform the experiments using xed levels
or random levels and perform the comparisons using the ANOVA test.
4
Metamodeling
A model can always be regarded as a transformation device which takes in values of the
independent (design) variables xi and transforms them into values of the output (response)
variable Y . A metamodel is a simplied approximation to the actual relationship between
inputs and output. A commonly used metamodel analysis methodology is regression analysis.
Consider a situation involving a single independent variable X and its associate dependent
variable Y . A simple linear regression model is
Y = + x+
where is a random error with mean zero and variance . For a total n pairs of observations
(xi; Yi ), the constants in the linear regression model are given by the normal equations
0
1
2
0
n Y
X
i
=
1
5
n
and
1
Pn Yi(xi x)
= Pn(x x)
i
1
2
1
Regression analysis must also test for signicance.
6