Download Air Travel Forecast Problem

Air Travel Forecast Problem
Objectives
• Introduction to forecasting methods
• Experience with Delphi
• Experience with consensus-seeking techniques
• Strength/weaknesses of various methods
Air Travel Forecast Problem
1
Methodology Tree for Forecasting
Knowledge
source
Judgmental
Others
Unstructured
Univariate
Self
Role
Structured
Multivariate
Data- Theorybased based
No role
Intentions/
expectations
Role playing
(Simulated
interaction)
Unaided
judgment
Statistical
Conjoint
analysis
Extrapolation
models
Quantitative
analogies
Data
mining
Neural
nets
Rule-based
forecasting
Feedback No feedback
Prediction
markets
Delphi
Structured
analogies
Linear
Game
theory
Air Travel Forecast Problem
Decomposition
Judgmental
bootstrapping
Expert
systems
Causal
models
Classification
Segmentation
Methodology Tree for Forecasting
forecastingpriciples.com
JSA-KCG
September 2005
2
Techniques for Forecasting
Form groups of about 5 to 7 people, then use the:
Delphi procedure
First estimate – individual and anonymous
Statistical summary – group
Group discussion (use consensus technique)
Second estimate – individual and anonymous
Statistical summary - group
Air Travel Forecast Problem
Minutes
12
3
20
2
3
40
3
Group Results
Accuracy
Rankings:
(Round 2)
Group
1
2
3
4
5
Average
ranks
Judgment
Bootstrapping
Segmentation
Causal model
Extrapolations
Air Travel Forecast Problem
4
Discussion
Discuss Delphi
Expected results
When to use
Actual Results
Initial hypotheses
Results in Air Travel study
Calculation of your error score
Conclusions
Air Travel Forecast Problem
5
Delphi
Agreement among experts
Your results
More agreement among panelists on Round 1
No differences (Round 1 vs. 2)
More agreement on Round 2
_____
_____
_____
Findings from literature: Typically more agreement on later rounds
Expected accuracy: Which do you expect to be closest to actual
ranks?
Your opinions
Round 1 more accurate
_____
Round 2 more accurate
_____
No difference
_____
Delphi improves accuracy vs. traditional meetings
given some expertise among panelists
Air Travel Forecast Problem
6
Round 2: Previous Rankings vs. Your Rankings
Average Ranking
MBA
(21 groups)*
Adv. Mgmt.
(28 groups)*
Judgment
2.2
2.4
Bootstrapping
3.2
2.9
Segmentation
2.2
2.0
Causal
2.6
2.9
Extrapolation
4.7
4.8
Method
You
*Groups from U.S., Sweden, Norway, and Netherlands
Air Travel Forecast Problem
7
Evidence-based Findings
(“>” means “more accurate than”)
1. Objective methods > subjective: especially for large
changes
2. Causal methods > naïve: especially for large changes
3. Bootstrapping > Judgment
4. Structured meetings > unstructured
Air Travel Forecast Problem
8
Using the Selection Tree
Sufficient
objective data
Judgmental methods
?
No
Quantitative methods
Yes
Good
knowledge of
relationships
Large changes
expected
No
Yes
Conflict among a few
decision makers
Policy analysis
No
Yes
No
Accuracy
feedback
Yes
No
Yes
Yes
Cross-section
Similar
cases exist
No
No
Yes
No
Judgmental
bootstrapping/
Decomposition
Large changes
likely
No
Time series
Yes
Good
domain
knowledge
Policy
analysis
Type of
knowledge
Domain
Yes
Type of
data
Policy
analysis
Unaided
judgment
Delphi/
Prediction
markets
No
Yes
Yes
No
Self
Conjoint
analysis
Intentions/
expectations
Role playing
(Simulated
interaction/
Game theory)
No
Single
method
Use unadjusted
forecast
Structured
analogies
Several
methods provide
useful forecasts
Omitted
information
?
No
Yes
Quantitative
analogies
Yes
Combine
forecasts
Use adjusted
forecast
Expert
systems
Rule-based
forecasting
Extrapolation/
Neural nets/
Data mining
Causal
models/
Segmentation
Selection Tree for Forecasting Methods
forecastingprinciples.com
JSA-KCG
January 2006
9
Rankings based on Evidence-based Findings
Method
Rank
Why?
Causal model
1.5
Segmentation
1.5
Extrapolation
3
Objective and naïve
Bootstrapping
4
Objective/subjective and causal
Judgment
5
Subjective and causal
Objective and causal
Evidence summarized in Armstrong (1985), Long-Range Forecasting, and Armstrong (2001), Principles of Forecasting –
see forecastingprinciples.com
Air Travel Forecast Problem
10
Accuracy of the Different Methods of Forecasting
U.S. Air Travel, 1963-1968
(Successive updating used)
Forecast Horizon
Years
Ahead
1
2
3
4
5
6
Averages
(Number
of Forecasts)
Mean Absolute Percentage Error*
Extrapolation
Judgment
Econometric
(6)
(5)
(4)
(3)
(2)
(1)
5.7
12.7
17.4
22.5
27.5
29.9
6.8
15.6
25.1
34.1
42.1
45.0**
4.2
6.8
7.3
9.8
6.2
0.7
(21)
19.3
28.1
5.8
* The forecasts were lower than actual in nearly all cases.
** Estimated
Source: Armstrong & Grohman (1972) in full text at forecastingprinciples.com
Air Travel Forecast Problem
11
Average Error Scores*
MBAs
Advanced Mgt.
Forecasting Experts
You
*Key:
Best possible
No information (all ties)
Worst possible
Air Travel Forecast Problem
Round 2
7.4
7.5
8.4
= 0
= 6
= 12
12
General Advice
• Beware of unaided judgment
• Be conservative when uncertain – thus, use equal ranks
given uncertainty about most accurate method
Air Travel Forecast Problem
13