Survey

* Your assessment is very important for improving the work of artificial intelligence, which forms the content of this project

Survey

Document related concepts

Transcript

RANKING AND SELECTION Choosing the Best among a set of Systems based on Simulation Output 1 R&S IS A VENERABLE PURSUIT Bechhofer, R. E., S. Elmaghraby and N. Morse. 1959. A single-sample multiple decision procedure for selecting the multinomial event which has the highest probability. Annals of Mathematical Statistics 30: 102-119. Bechhofer, R. E. and D. Goldsman. 1986. Truncation of the Bechhofer-Jiefer-Sobel sequential procedure for selecting the multinomial event with the largest probability (II): extended tables and improved procedure. Communications in Statistics – Simulation and Computation B15:829-851. Many papers in the ’90’s examining Variance Reduction Techniques within this context 2 MISSION • Choose the Best alternative • Know the probability of being incorrect • Know how big to make the sample to ensure a specified probability of correct selection • Know the effect of being picky (what if the chosen system performs within d of the best (unknown) system?) 3 REQUIRED ASSETS • “Indifference zone” size d • Specified P[wrong selection] = a • Procedure to follow – Often involves tables of values developed using complex probabilistic methods or massive simulation datasets • Flexible simulation 4 TYPICAL SET-UP • Given n alternate choices • Select the choice that has the greatest Expected Performance 5 TWO-STAGE RINOTT • Stage 1: Baseline – take i.i.d. samples Yi1, Yi2, ..., Yin0 from sim. of configuration i – estimate si2 for system i – look up h from a Rinott Table using a and d – Ni=max{n0, (hsi/d)2} 6 ...more RINOTT • Stage 2 – Sample Ni-n0 more from system i – compute Y-bar for each system – Pick the superior system • Rinott also has a confidence interval for runner-up’s performance 7 ...discussing RINOTT • h is a scale parameter reflecting both strictures on precision and accuracy • n0 is recommended to be >= 10 – bigger gives better s2 and potentially greater waste • Simulation must be easily restarted 8 TWO-STAGE RINOTT SCREENING PROCEDURE • Goal: Choose a subset of specified size m containing the best choice with probability 1-a. • Similar feel, different table. • Used to screen large groups of nominees 9 MULTINOMIAL SELECTION PROBLEM • Yi,j is the jth sample from the ith system • pi=Prob[system i produces the largest value in {Y1,j, Y2,j,..., Ys,j}] (unknown) • Choose highest p • Indifference: q= ratio of highest p to next highest 10 SINGLE-STAGE BECHHOFER-ELMAGHRABYMORSE (BEM) • Using a and q, we can look up n from a Bechhofer table • Hold n “contests” comparing {Y1,j, Y2,j,..., Ys,j} • Pick the system that wins most often 11 ...discussing MULTINOMIAL SELECTION • P[beat the field] not the same as having the greatest Expected Value • Selecting a best-to-buy • Selecting a favorite golfer • Remember the Turtles! – the distribution of the Winner is not the distribution of any one contestants 12 BECHHOFER-GOLDSMAN • Using a and q, look up nT, the truncation sample size • After contest m, let W[i] be the number of wins of the ith winningest system • Let Z = (1 / q ) W[1] W[ i ] 13 B-Goldsman • Stop and declare a winner when – Z < a/(1a) – concluded nT contests – there are no longer W[1]-W[2] trials left before we reach nT 14 ...discussing BGoldsman • Sample exactly once for each system for each round • Very stingy with samples • Randomize in the case of ties 15 COMPARISON PROCEDURES • Compare a set of systems with an existing system with known expected value • Compare a set of systems with a default stochastic system • Pick the best system based on another characteristic (a-quantile, IQR) • Common random numbers • Steady-state simulations • Tables on p. 606-607 in 2nd Edition 16