Download Teaching Spreadsheet Simulation

Teaching Simulation
Roger Grinde, [email protected]
University of New Hampshire
Files: http://pubpages.unh.edu/~rbg/TMS/TMS_Support_Files.html
Teaching Simulation

Do you teach simulation?





In which courses?
With spreadsheets? Add-Ins?
Monte Carlo? Discrete Event?
Do you use simulation to help teach other
topics?
Do other courses at your school use
simulation?
Session Overview








Common Student Misunderstandings
Simulation-Related Learning Goals
Motivations
Building on Other Methodologies
Effects of Correlation
Interpreting Results
Software Issues
Considerations, Recommendations
Student Misunderstandings


What are some misunderstandings
students have about decision-making in
the face of uncertainty?
What are some common errors students
make in simulation?
Some Considerations

Decide which learning goals are most important,
and structure coverage so those goals are
attained.






Student backgrounds
Time constraints
Overall course objectives
Inter-course relationships, role of course in curriculum
Monte-Carlo and/or Discrete-Event? Related
software selection question.
Teaching environment, class size, TA support,
etc.
Learning Goals

What are your learning goals when
teaching simulation?





Fundamental Concepts
Methodology of Simulation
Applications of Simulation
Modeling Knowledge & Skills
Critical & Analytical Thinking
Simulation, Statistidcal,
Spreadsheet Modeling,
Decision Making Concepts
Mapping: Learning Goals to
Examples
Examples/Application Areas
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Games/Tournaments
X
X
Personal Financial
Planning
Multiple Project
Selection
Stock Price
Modeling, Option
Pricing
Inventory (multiperiod)
X
X
X
X
Queuing
X
Capital Project NPV
Extension of other analaysis tools
Is simulation needed?
Variety of probability distributions
Model-building issues (where a simulation model would be
different than a deterministic model)
Output distribution as function of input distributions
Historical/empirical data
Summary statistics
Alternate decision criteria & risk measures
Sources of error
Correlation and/or relationships among input variables
Optimization concepts in simulation
Portfolio Allocation
Learning Goal/Objective
Inventory (singleperiod)
Mapping: Goals to Examples
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Motivations (Why is simulation
useful?)

Two investment alternatives

A: Invest $10,000.



B: Invest $10,000



Probability of a $100,000 gain is 0.10
Probability of a $10,000 loss is 0.90
Probability of a $500 gain is 1.0
Which would you choose?
Why?
Risk-Informed Decision Making




Appropriate and inappropriate uses of averages.
Managers manage risk.
Simulation gives us a tool to help us evaluate risk.
Risk: The uncertainty associated with an undesirable
outcome.


Risk is not the same as just being uncertain about
something, and is not just the possibility of a bad outcome.
Risk considers the likelihood of an undesirable outcome
(e.g., the probability) as well as the magnitude of that
outcome.
“Flaw of Averages” (Sam Savage)

Article by Sam Savage
(http://www.stanford.edu/~savage/faculty/savage/)

Annuity Illustration (historical simulation)
Simulation Model Schematic
Fixed (Known) Inputs
Random (Uncertain) Inputs
Simulation Model
Outputs &
Performance Measures
Decision Variables

Concept of an output “distribution.”
Foundations of Simulation



Randomness, Uncertainty
Probability Distributions
Tools



Dice Roller (John Walkenbach:
http://www.j-walk.com/ss)
Die Roller (modified)
Interactive Simulation Tool
Extending Other
Methodologies






Spreadsheet Engineering
Base Case Analysis
What-If Analysis, Scenario Analysis
Critical Value Analysis
Sensitivity Analysis
Simulation
Extending Other
Methodologies





Familiar Example/Case; Students have already
developed model and done some deterministic
analysis.
Students provided with some probability
distribution information
Develop comfort with mechanics of simulation
See the “value added” of simulation
Provides entry point for discussion of important
questions
Example: Watson Truck



Adapted from Lawrence & Weatherford
(2001)
Students have previously built basecase model, done “critical value”
analysis (using Goal Seek), and have
done sensitivity analysis (data tables,
tornado charts)
Link to files: PDF, Sensitivity, Simulation
Watson Truck: Inputs
A
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
B
C
D
E
F
G
Watson Truck Rental
Parameters/Uncontrollable Inputs
Purchase Price
Property Tax
Var Cost/Truck
# Trucks
Prop Tax Growth
Truck Cost Growth
Base Truck Rental Rate
% Trucks Rented @ $1000
Rental Rate Slope
Rental Rate Inflation
Business Sale Multiplier
Discount Rate
$1,000,000
$35,000
$4,800
50
4%
7%
$1,000
60%
7%
9%
3
10.0%
Decision Variable
Rental Rate (decision variable)
$1,000
Intermediate Calculations
Rental Rate Slope
% Trucks Rented
-0.07%
60.0%
per year
per year
per month
per $100 reduction in rental rate
Sales price assumed to be 3*(year 3 revenues)
per $1 increase in rental rate
Watson Truck: Base Case
Model
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
A
B
Primary Output
Net Present Value (@ discount rate)
C
D
E
F
G
H
C25: =NPV(C15,D40:F40)+C40
$209,769
Cash Flow Model
0
Cash Inflows
Truck Rental Income
Business Sale
Total Inflows
Cash Outflows
Purchase Price
Property Tax
Truck Var Cost
Total Outflows
1
2
3
$360,000
$392,400
$360,000
$392,400
$427,716
$1,283,148
$1,710,864
$1,000,000
$35,000
$240,000
$275,000
$36,400
$256,800
$293,200
$37,856
$274,776
$312,632
($1,000,000)
$85,000
$99,200
$1,398,232
F30: =E30*(1+$C13)
$0
$1,000,000
F31: =C14*F30
F32: =SUM(F30:F31)
F36: =E36*(1+$C8)
F37: =E37*(1+$C9)
F38: =SUM(F35:F37)
F40: =F32-F38
Net Cash Flow
Watson Truck: Sensitivity
Analysis
Tornado Sensitivity Chart
Output Measure
$0
$50,000 $100,000 $150,000 $200,000 $250,000 $300,000 $350,000 $400,000 $450,000 $500,000
Base Truck Rental Rate
% Trucks Rented @ $1000
# Trucks
Parameter
Purchase Price
Business Sale Multiplier
Var Cost/Truck
Discount Rate
Rental Rate Inflation
Rental Rate Slope
Property Tax
Truck Cost Growth
Prop Tax Growth
-10 Pct
+10 Pct
Watson: Simulation
Forecast: Net Present Value (@ discount rate)
1,000 Trials
Frequency Chart
1,000 Displayed
.035
35
.026
26.25
.018
17.5
.009
8.75
Mean = $232,119
.000
0
($525,250)
($139,570)
$246,111
$631,792
Certainty is 82.00% from $0 to +Infinity Dollars
$1,017,472
Learning Goals Addressed (at
least partially)








Linkage with other course/functional area
What inputs should we simulate?
Useful probability distributions. Choice of
parameters. Subjective versus objective
estimates.
Concept of an output distribution
What results are important?
Sources of error in simulation
Simulation mechanics
Simulation in context with other tools
Example: Single-Period
Portfolio

Simple example, but helps address a number of
learning goals







Do we need to simulate?
Effect of correlation among input quantities
Confidence vs. Prediction (certainty) intervals
Quantification of risk, multiple decision criteria
Optimization concepts within simulation context
Precision of estimates from simulation
Link to file
Spreadsheet
B
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
C
D
E
Portfolio Allocation Model
Investments
Money Market fund
Income fund
Growth and Income fund
Aggressive Growth fund
Total amount available
Annual
return
3.0%
5.0%
7.0%
11.0%
$100,000
Decision variables
Money Market fund
Income fund
Growth and Income fund
Aggressive Growth fund
Total expected return
Amount
invested
$25,000
$25,000
$25,000
$25,000
$6,500
Lower
bound
$0
$10,000
$0
$10,000
Upper
bound
$50,000
$25,000
$80,000
$100,000
Do we need simulation?

Assuming we know the distributions for
the returns, do we need simulation to
compute the



expected return of the portfolio?
variance of the portfolio?
tail probabilities?
What if the asset returns are
correlated?

Large Stocks
Large Growth Stocks
Large Value Stocks
Small Stocks
Small Growth Stocks
Small Value Stocks
Foreign Stocks
Bonds
What is the effect of correlation on the
distribution of portfolio returns?
Large
Stocks
Large
Growth
Stocks
Large
Value
Stocks
1
0.958411
0.901159
0.720036
0.755265
0.507037
0.391551
0.366054
1
0.74063
0.606356
0.720758
0.322101
0.294704
0.267404
1
0.776748
1
0.678815 0.921266
1
0.715396 0.875155 0.618755
1
0.454882 0.275857 0.325624 0.140232
1
0.465424 0.28663 0.164181 0.369522 0.112362
Small
Stocks
Small
Growth
Stocks
Small
Value
Stocks
Foreign
Stocks
Based on Standard & Poor Micropal, via Franklin/Templeton Investor Topics Update, Winter 2001
(Asset Returns from 1980-2000)
Bonds
1
Results (n=1000)

No Correlation




Mean = $6842
Standard Deviation = $5449
5% VaR = ($2165)
Positive Correlation



Mean = $6409
Standard Deviation = $7386
5% VaR = ($5655)
Decision Criteria, Risk
Measures


What criteria are important for making decision
as to where to invest? Average? Standard
Deviation? Minimum? Maximum? Quartiles? VaR?
Probability of Loss?
Measures of risk.


Simulation gives us the entire output distribution.
Entry point for optimization within simulation
context

Alternate scenarios, efficient frontier, OptQuest,
RiskOptimizer, etc.
Confidence Intervals



Students can (usually) calculate a
confidence interval for the mean.
Do they know what it means?
Reconciling confidence and prediction
intervals.
Sample Results (Portfolio
Problem)
Statistics:
Trials
Mean
Median
Mode
Standard Deviation


90% Confidence Interval
Standard Error
$233.56
Z
1.645
Lower Limit
$6,025
Upper Limit
$6,794
90% CI on Mean Dollar Return: ($6025, $6794)
What does that confidence interval mean?


Value
1000
$6,409
$6,531
--$7,386
Common (student) error
What does the CI about an individual outcome? For
example, from this year’s return?
Sample Results (cont)
Percentile
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
50%
55%
60%
65%
70%
75%
80%
85%
90%
95%
100%
dollars
($16,088)
($5,655)
($3,052)
($1,008)
$271
$1,277
$2,498
$3,484
$4,307
$5,365
$6,531
$7,695
$8,349
$9,310
$10,386
$11,419
$12,431
$13,896
$15,689
$18,659
$30,330



Cumulative Percentiles of
the Portfolio Return
Distribution
What do these results
mean?
What is the 90%
“prediction” (or “certainty”)
interval (centered around
the median)?
Putting Them Together

90% Confidence Interval for the Mean


90% Prediction Interval (centered around
median)



($6025, $6794)
(-$5655, $18,659)
Note: Crystal Ball uses the term “certainty”)
Students:


Understand the difference?
Understand when one is more appropriate than
the other?
Precision of Simulation Results


Since we know the true value of the
mean (for the portfolio problem), this
can be a good example to look at
precision and sample size issues.
Confidence interval for proportion or for
a given percentile sometimes makes
more sense.
Crystal Ball: Precision Control


Nice way to illustrate effect of sample size.
Precision Control stops simulation based on
user-specified precision on the mean,
standard deviation, and/or a percentile.



Actually, CB stops whenever the first of a number
of conditions occurs (e.g., maximum number of
trials, precision specifications).
Example (Portfolio Allocation)
Example (Option Pricing)
Precision: Portfolio Example
Precision Control Experiment: Summary
Number of Trials Required for Specified Precision
Precision
$800
$400
$200
$100
Mean
300
1250
5350
21,700
5th Percentile
1900
4950
20,300
95,200
Note: "Precision" is the half-width of the 95% confidence interval
True Mean = $6,500
Precision: Option Pricing
Example
Precision Control Experiment: Mean Call and Put Values
Number of Trials Required for Specified Precision of Mean Call and Put Values
Trials
50
200
700
2600
64400
Precision: Call Precision: Put
$0.259
$0.390
$0.166
$0.181
$0.099
$0.090
$0.050
$0.047
$0.010
$0.009
Note: "Precision" is the half-width of the 95% confidence interval
Black-Scholes Call Price = $0.733
Black-Scholes Put Price = $0.955
Crystal Ball Functions and
Simple VBA Control

Crystal Ball provides built-in functions



Control through VBA



Distribution Functions (e.g., CB.Normal)
Functions for Accessing Simulation Results (e.g.,
CB.GetForeStatFN)
For some students, can be a hook into greater interest
in simulation and/or VBA/DSS.
Allows one to prepare a simulation-based model for
someone who doesn’t know Crystal Ball.
Example
VBA-Enabled Example
B
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
C
D
E
F
G
H
I
J
Portfolio Allocation Model
Investments
Money Market fund
Income fund
Growth and Income fund
Aggressive Growth fund
Total amount available
Annual
return
3.5%
4.1%
6.5%
29.3%
$100,000
Decision variables
Money Market fund
Income fund
Growth and Income fund
Aggressive Growth fund
Total return
Amount
invested
$25,000
$25,000
$25,000
$25,000
$10,866
Param1
2.0%
5.0%
7.0%
11.0%
Distributions/Parameters
Param2
Distribution
4.0%
Uniform
5.0%
Normal
12.0%
Normal
18.0%
Normal
C5: =CB.Uniform(D5,E5)
C6: =CB.Normal(D6,E6) (copied down)
$C$17: =SUMPRODUCT(C5:C8,C13:C16)
(Forecast Cell)
C22: =CB.GetForeStatFN(C$17,1)
Simulation Results
Summary Statistics
Number Trials
Mean
Standard Deviation
Minimum
Maximum
Standard Error of Mean
400
$6,695
$5,644
($9,718)
$20,612
$282
Value at Risk (enter %)
Critical Value (enter $)
5%
$4,000
D30: =CB.GetCertaintyFN(C$17,C30)/100
Quartiles
0%
25%
50%
75%
100%
($2,985)
0.305
($9,718)
$2,840
$6,931
$10,730
$20,612
5.0% chance return will be <= ($2,985)
Probability (Return <= $4,000) = 0.305
F22: =CB.GetForePercentFN(C$17,E22*100)
Enter Number of Trials
400
Run Simulation
CB. Functions and VBA

CB. Distribution Functions


CB. Functions for reporting results


e.g., CB.Normal, CB.Uniform, CB.Triangular)
CB.GetForeStatFN, CB.GetCertaintyFN, CB.GetForePercentFN
VBA: simple to automate specific processes
Sub RunSimulation()
CB.ResetND
CB.Simulation Range("n_trials").Value
End Sub
Sub CreateReport()
CB.CreateRpt
' CB.CreateRptND cbrptOK
End Sub
Learning Goals Revisited

Decide which learning goals are the most
important, and structure coverage so those
goals are attained.




Student backgrounds
Time constraints
Overall course objectives
Mapping of learning goals to examples,
cases, and projects that you will use.
Simulation, Statistidcal,
Spreadsheet Modeling,
Decision Making Concepts
Mapping: Learning Goals to
Examples
Examples/Application Areas
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Games/Tournaments
X
X
Personal Financial
Planning
Multiple Project
Selection
Stock Price
Modeling, Option
Pricing
Inventory (multiperiod)
X
X
X
X
Queuing
X
Capital Project NPV
Extension of other analaysis tools
Is simulation needed?
Variety of probability distributions
Model-building issues (where a simulation model would be
different than a deterministic model)
Output distribution as function of input distributions
Historical/empirical data
Summary statistics
Alternate decision criteria & risk measures
Sources of error
Correlation and/or relationships among input variables
Optimization concepts in simulation
Portfolio Allocation
Learning Goal/Objective
Inventory (singleperiod)
Mapping: Possible Learning Goals to
Examples
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
Common Student Errors






Thinking of simulation as the method of first choice.
Simulating too many quantities.
Too much focus on distribution/parameter selection or on
the numerical results, not enough on insights/decision.
Misinterpretation of results, especially confidence intervals
Modeling: Using same return, lead time, etc. for every time
period/order, etc. (difference between deterministic and
simulation models)
Choosing the assumptions, distributions, parameters, etc.
that give the “best” numerical results.
Software Issues: Monte-Carlo

Alternatives





“Full-Service” Add-In? (e.g., @Risk, Crystal Ball, XLSim
by Sam Savage, RiskSim)
“Helper” Workbook? (e.g., Interactive Simulation Tool
with Random Number Function support)
“Native” Excel?
All have advantages, disadvantages
Back to learning objectives, role of course,
student audience, etc.
Software Issues: DiscreteEvent

Alternatives




Stand-alone package (e.g., Arena, Process Model,
Extend)
Excel Add-In (e.g., SimQuick by David Hartvigsen)
Native Excel modeling augmented by Monte Carlo
tool (e.g., QueueSimon by Armann Ingolfsson)
DE Simulation can be a great way to help teach
concepts in other areas (e.g., queuing,
inventory)

Don’t necessarily need to teach DE Simulation to be
able to use it to teach other things.
Other Considerations






Program-level, inter-course objectives
Role of course in curriculum
Level/background of students
Monte-Carlo and/or Discrete-Event? Related
software selection question.
Teaching environment, class size, TA support,
etc.
How much of course can/should be devoted to
simulation?
Recommendations



Learning Goals: Figure out what you really want
students to learn and be able to do, after your
class is over; in other classes, internships, future
jobs? How can simulation coverage help
accomplish these goals?
Cases: Engage students in the business problem,
let them discover relevance of simulation.
Student-Developed Projects: Students gain better
awareness of all the “little” decisions involved in
modeling and simulation.
Additional Slides
Concept Coverage Through
Examples



Philosophy: Expose students to a number of
application areas, but at the same time covering
fundamental decision-making, modeling, and
analysis concepts and methodologies.
Counter to the way many of us were taught.
Key: We need to clearly understand which
concepts we’re trying to convey with each
example.
Examples that Work Well




Fundamentals: Dice Roller, Interactive Simulation
Tool
Personal Decisions: Car Repair/Purchase Decision,
Portfolio (single period, based on CB Model), College
Funding (based on Winston & Albright)
Capital Project Evaluation: Truck Rental Company
(based on Lawrence & Weatherford), Project
Selection/Diversification (CB Model), Product
Development & Launch (CB Model)
Finance: Stock Price Models, Option Pricing,
Random Walks, Mean Reverting Processes
Examples (continued)





Inventory: DG Winter Coats (NewsVendor),
Antarctica (multi-period, based on Lapin & Whisler)
Queuing: QueueSimon (Armonn Ingolfsson)
Games/Tournaments, Sports: NCAA Tourney
(based on Winston & Albright), Home Run Derby
Baseball Simulation (VBA-enabled), Baseball Inning
Simulation
Simulation in Teaching Other Topics: Revenue
Management Illustration, QueueSimon (Armonn
Ingolfsson)
Crystal Ball Features: CB Macros, CB Functions
Examples Posing Difficulties
for Spreadsheets




Multi-server queues and queue
networks
Most production systems
Business process redesign
However, some add-ins do exist for
simple discrete-event models (e.g.,
SimQuick by David Hartvigsen)
Sources of Error in Simulation

What are some of the sources of error
in a spreadsheet simulation
model/analysis?
Learning Objectives
(Revisited)





General
Probability Distributions
Statistics
Relationships Among Variables
Decision Making
Possible Learning Goals

General




Use simulation as an extension of other analysis tools
Apply simulation to a variety of business problems
Identify when simulation is and is not needed to analyze a
situation
Probablilty Distributions



Understand and use probability distributions to model
phenomena
Describe the output distribution, understanding this to be a
function of the input distributions
Use historical/empirical data and subjective assessments
appropriately in choosing distributions and parameters
Possible Learning Goals (cont)

Statistics




Correctly interpret summary statistics, including
percentiles/histograms
Correctly interpret confidence and prediction (certainty)
intervals
Identify sources of error in simulation, apply to specific
situations
Relationships Among Variables


Include appropriate correlation and/or other relationships
when model building
Describe the effect of correlation and/or other relationship
on simulation results
Possible Learning Goals (cont)

Decision Making



Identify and correctly use different risk
measures
Use appropriate criteria in making
recommendations
Use optimization concepts in a simulation
application
Student Project Example
(MBA)


PPT File
Excel File