Survey

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Document related concepts

History of statistics wikipedia, lookup

Statistics wikipedia, lookup

Probability wikipedia, lookup

Transcript
```RANKING AND SELECTION
Choosing the Best among a
set of Systems based on
Simulation Output
1
R&S IS A VENERABLE
PURSUIT
Bechhofer, R. E., S. Elmaghraby and N. Morse.
1959. A single-sample multiple decision
procedure for selecting the multinomial event
which has the highest probability. Annals of
Mathematical Statistics 30: 102-119.
Bechhofer, R. E. and D. Goldsman. 1986.
Truncation of the Bechhofer-Jiefer-Sobel
sequential procedure for selecting the
multinomial event with the largest probability
(II): extended tables and improved procedure.
Communications in Statistics – Simulation and
Computation B15:829-851.
Many papers in the ’90’s examining Variance
Reduction Techniques within this context
2
MISSION
• Choose the Best alternative
• Know the probability of being
incorrect
• Know how big to make the sample
to ensure a specified probability
of correct selection
• Know the effect of being picky
(what if the chosen system
performs within d of the best
(unknown) system?)
3
REQUIRED ASSETS
• “Indifference zone” size d
• Specified P[wrong selection] = a
• Procedure to follow
– Often involves tables of values
developed using complex probabilistic
methods or massive simulation
datasets
• Flexible simulation
4
TYPICAL SET-UP
• Given n alternate choices
• Select the choice that has
the greatest Expected
Performance
5
TWO-STAGE RINOTT
• Stage 1: Baseline
– take i.i.d. samples Yi1, Yi2,
..., Yin0 from sim. of
configuration i
– estimate si2 for system i
– look up h from a Rinott Table
using a and d
– Ni=max{n0, (hsi/d)2}
6
...more RINOTT
• Stage 2
– Sample Ni-n0 more from system i
– compute Y-bar for each system
– Pick the superior system
• Rinott also has a confidence
interval for runner-up’s
performance
7
...discussing RINOTT
• h is a scale parameter
reflecting both strictures on
precision and accuracy
• n0 is recommended to be >= 10
– bigger gives better s2 and
potentially greater waste
• Simulation must be easily
restarted
8
TWO-STAGE RINOTT
SCREENING PROCEDURE
• Goal: Choose a subset of
specified size m containing
the best choice with
probability 1-a.
• Similar feel, different
table.
• Used to screen large groups
of nominees
9
MULTINOMIAL SELECTION
PROBLEM
• Yi,j is the jth sample from the
ith system
• pi=Prob[system i produces the
largest value in {Y1,j, Y2,j,...,
Ys,j}]
(unknown)
• Choose highest p
• Indifference: q= ratio of highest
p to next highest
10
SINGLE-STAGE
BECHHOFER-ELMAGHRABYMORSE (BEM)
• Using a and q, we can look up
n from a Bechhofer table
• Hold n “contests” comparing
{Y1,j, Y2,j,..., Ys,j}
• Pick the system that wins
most often
11
...discussing
MULTINOMIAL SELECTION
• P[beat the field] not the same as
having the greatest Expected
Value
• Selecting a favorite golfer
• Remember the Turtles!
– the distribution of the Winner is not
the distribution of any one
contestants
12
BECHHOFER-GOLDSMAN
• Using a and q, look up nT, the
truncation sample size
• After contest m, let W[i] be
the number of wins of the ith
winningest system
• Let
Z =  (1 / q )
W W[ i ]
13
B-Goldsman
• Stop and declare a winner
when
– Z < a/(1a)
– concluded nT contests
– there are no longer W-W
trials left before we reach nT
14
...discussing BGoldsman
• Sample exactly once for each
system for each round
• Very stingy with samples
• Randomize in the case of ties
15
COMPARISON PROCEDURES
• Compare a set of systems with an
existing system with known expected
value
• Compare a set of systems with a
default stochastic system
• Pick the best system based on another
characteristic (a-quantile, IQR)
• Common random numbers