Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
RANKING AND SELECTION Choosing the Best among a set of Systems based on Simulation Output 1 R&S IS A VENERABLE PURSUIT Bechhofer, R. E., S. Elmaghraby and N. Morse. 1959. A single-sample multiple decision procedure for selecting the multinomial event which has the highest probability. Annals of Mathematical Statistics 30: 102-119. Bechhofer, R. E. and D. Goldsman. 1986. Truncation of the Bechhofer-Jiefer-Sobel sequential procedure for selecting the multinomial event with the largest probability (II): extended tables and improved procedure. Communications in Statistics – Simulation and Computation B15:829-851. Many papers in the ’90’s examining Variance Reduction Techniques within this context 2 MISSION • Choose the Best alternative • Know the probability of being incorrect • Know how big to make the sample to ensure a specified probability of correct selection • Know the effect of being picky (what if the chosen system performs within d of the best (unknown) system?) 3 REQUIRED ASSETS • “Indifference zone” size d • Specified P[wrong selection] = a • Procedure to follow – Often involves tables of values developed using complex probabilistic methods or massive simulation datasets • Flexible simulation 4 TYPICAL SET-UP • Given n alternate choices • Select the choice that has the greatest Expected Performance 5 TWO-STAGE RINOTT • Stage 1: Baseline – take i.i.d. samples Yi1, Yi2, ..., Yin0 from sim. of configuration i – estimate si2 for system i – look up h from a Rinott Table using a and d – Ni=max{n0, (hsi/d)2} 6 ...more RINOTT • Stage 2 – Sample Ni-n0 more from system i – compute Y-bar for each system – Pick the superior system • Rinott also has a confidence interval for runner-up’s performance 7 ...discussing RINOTT • h is a scale parameter reflecting both strictures on precision and accuracy • n0 is recommended to be >= 10 – bigger gives better s2 and potentially greater waste • Simulation must be easily restarted 8 TWO-STAGE RINOTT SCREENING PROCEDURE • Goal: Choose a subset of specified size m containing the best choice with probability 1-a. • Similar feel, different table. • Used to screen large groups of nominees 9 MULTINOMIAL SELECTION PROBLEM • Yi,j is the jth sample from the ith system • pi=Prob[system i produces the largest value in {Y1,j, Y2,j,..., Ys,j}] (unknown) • Choose highest p • Indifference: q= ratio of highest p to next highest 10 SINGLE-STAGE BECHHOFER-ELMAGHRABYMORSE (BEM) • Using a and q, we can look up n from a Bechhofer table • Hold n “contests” comparing {Y1,j, Y2,j,..., Ys,j} • Pick the system that wins most often 11 ...discussing MULTINOMIAL SELECTION • P[beat the field] not the same as having the greatest Expected Value • Selecting a best-to-buy • Selecting a favorite golfer • Remember the Turtles! – the distribution of the Winner is not the distribution of any one contestants 12 BECHHOFER-GOLDSMAN • Using a and q, look up nT, the truncation sample size • After contest m, let W[i] be the number of wins of the ith winningest system • Let Z = (1 / q ) W[1] W[ i ] 13 B-Goldsman • Stop and declare a winner when – Z < a/(1a) – concluded nT contests – there are no longer W[1]-W[2] trials left before we reach nT 14 ...discussing BGoldsman • Sample exactly once for each system for each round • Very stingy with samples • Randomize in the case of ties 15 COMPARISON PROCEDURES • Compare a set of systems with an existing system with known expected value • Compare a set of systems with a default stochastic system • Pick the best system based on another characteristic (a-quantile, IQR) • Common random numbers • Steady-state simulations • Tables on p. 606-607 in 2nd Edition 16