Download SELECTION

RANKING AND SELECTION
Choosing the Best among a
set of Systems based on
Simulation Output
1
R&S IS A VENERABLE
PURSUIT
Bechhofer, R. E., S. Elmaghraby and N. Morse.
1959. A single-sample multiple decision
procedure for selecting the multinomial event
which has the highest probability. Annals of
Mathematical Statistics 30: 102-119.
Bechhofer, R. E. and D. Goldsman. 1986.
Truncation of the Bechhofer-Jiefer-Sobel
sequential procedure for selecting the
multinomial event with the largest probability
(II): extended tables and improved procedure.
Communications in Statistics – Simulation and
Computation B15:829-851.
Many papers in the ’90’s examining Variance
Reduction Techniques within this context
2
MISSION
• Choose the Best alternative
• Know the probability of being
incorrect
• Know how big to make the sample
to ensure a specified probability
of correct selection
• Know the effect of being picky
(what if the chosen system
performs within d of the best
(unknown) system?)
3
REQUIRED ASSETS
• “Indifference zone” size d
• Specified P[wrong selection] = a
• Procedure to follow
– Often involves tables of values
developed using complex probabilistic
methods or massive simulation
datasets
• Flexible simulation
4
TYPICAL SET-UP
• Given n alternate choices
• Select the choice that has
the greatest Expected
Performance
5
TWO-STAGE RINOTT
• Stage 1: Baseline
– take i.i.d. samples Yi1, Yi2,
..., Yin0 from sim. of
configuration i
– estimate si2 for system i
– look up h from a Rinott Table
using a and d
– Ni=max{n0, (hsi/d)2}
6
...more RINOTT
• Stage 2
– Sample Ni-n0 more from system i
– compute Y-bar for each system
– Pick the superior system
• Rinott also has a confidence
interval for runner-up’s
performance
7
...discussing RINOTT
• h is a scale parameter
reflecting both strictures on
precision and accuracy
• n0 is recommended to be >= 10
– bigger gives better s2 and
potentially greater waste
• Simulation must be easily
restarted
8
TWO-STAGE RINOTT
SCREENING PROCEDURE
• Goal: Choose a subset of
specified size m containing
the best choice with
probability 1-a.
• Similar feel, different
table.
• Used to screen large groups
of nominees
9
MULTINOMIAL SELECTION
PROBLEM
• Yi,j is the jth sample from the
ith system
• pi=Prob[system i produces the
largest value in {Y1,j, Y2,j,...,
Ys,j}]
(unknown)
• Choose highest p
• Indifference: q= ratio of highest
p to next highest
10
SINGLE-STAGE
BECHHOFER-ELMAGHRABYMORSE (BEM)
• Using a and q, we can look up
n from a Bechhofer table
• Hold n “contests” comparing
{Y1,j, Y2,j,..., Ys,j}
• Pick the system that wins
most often
11
...discussing
MULTINOMIAL SELECTION
• P[beat the field] not the same as
having the greatest Expected
Value
• Selecting a best-to-buy
• Selecting a favorite golfer
• Remember the Turtles!
– the distribution of the Winner is not
the distribution of any one
contestants
12
BECHHOFER-GOLDSMAN
• Using a and q, look up nT, the
truncation sample size
• After contest m, let W[i] be
the number of wins of the ith
winningest system
• Let
Z =  (1 / q )
W[1] W[ i ]
13
B-Goldsman
• Stop and declare a winner
when
– Z < a/(1a)
– concluded nT contests
– there are no longer W[1]-W[2]
trials left before we reach nT
14
...discussing BGoldsman
• Sample exactly once for each
system for each round
• Very stingy with samples
• Randomize in the case of ties
15
COMPARISON PROCEDURES
• Compare a set of systems with an
existing system with known expected
value
• Compare a set of systems with a
default stochastic system
• Pick the best system based on another
characteristic (a-quantile, IQR)
• Common random numbers
• Steady-state simulations
• Tables on p. 606-607 in 2nd Edition
16