Download Talk

Verification of nowcasts and
very short range forecasts
Beth Ebert
BMRC, Australia
WWRP Int'l Symposium on Nowcasting and Very Short Range Forecasting, Toulouse, 5-9 Sept 2005
1
Why verify forecasts?
• To monitor performance over time
 summary scores
• To evaluate and compare forecast
systems
 continuous and
categorical scores
• To show impact of forecast
 skill & value scores
• To understand error in order to
improve forecast system
 diagnostic methods
The verification approach taken depends on the purpose of the
verification
2
Verifying nowcasts and very short range forecasts
Nowcast characteristics
Impact on verification
concerned mainly with high impact
weather
rare events difficult to verify in
systematic manner
may detect severe weather
elements
storm spotter observations &
damage surveys required
observations-based
same observations often used to
verify nowcasts
high temporal frequency
many nowcasts to verify
high spatial resolution
observation network usually not
dense enough (except radar)
small spatial domain
relatively small number of standard
observations
3
Observations – issues for nowcasts
Thunderstorms and severe weather (mesocyclones, hail,
lightning, damaging winds)
• Spotter observations may contain error
• Biased observations
• More observations during daytime & in populated areas
• More storm reports when warnings were in effect
• Cell mis-association by cell tracking algorithms
Precipitation
• Radar rain rates contain error
• Scale mismatch between gauge observations and radar pixels
Observation error can be large but is usually neglected
 more research required on handling observation error
4
Matching forecasts and observations
• Matching approach depends on
• Nature of forecasts and observations
• Scale
• Consistency
• Sparseness
• Other matching criteria
point-to-grid
grid-to-point
Forecast grid
• Verification goals
• Use of forecasts
• Matching approach can impact
verification results
• Grid to grid approach
Observed grid
• Overlay forecast and observed grids
• Match each forecast and observation
5
Forecast Quality Definitions
Wilson subjective categories
1 – forecast and observed almost perfect overlap.
2 – majority of observed and forecast echoes overlap or offsets <50 km
3 – forecast and observed look similar but there are a number of echo
offsets and several areas maybe missing or extra.
4 – the forecasts and observed are significantly different with very little
overlap; but some features are suggestive of what actually occurred.
5 – there is no resemblance to forecast and observed.
First rule of forecast verification – look at the results!
Systematic verification – many cases
Aggregation and stratification
• Aggregation
•
•
•
•
More samples  more robust statistics
Across time - results for each point in space
Space - results for each time
Space and time - results summarized across spatial region
and across time
• Stratification
• Homogeneous subsamples  better understanding of how
errors depend on regime
• By location or region
• By time period (diurnal or seasonal variation)
7
Real-time nowcast verification
• Rapid feedback from latest radar scan
• Evaluate the latest objective guidance while it is still "fresh"
• Better understand strengths and weaknesses of nowcast system
• Tends to be subjective in nature
1
radar anal
LAPS05
Probability of exceedance
0.1
0.01
0.001
0.0001
0.00001
0.000001
0
10
20
30
40
50
60
70
Rain (mm/h)
Real time forecast verification system (RTFV) under development in BMRC
• Not commonly performed!
8
Post-event verification
POD
• More observations may be available
• verification results more robust
• No single measure is adequate!
Frequency
bias
• several metrics needed
• distributions-oriented verification
• scatter plots
• (multi-category) contingency tables
• box-whisker plots
CSI
FAR
• Confidence intervals recommended,
especially when comparing one set
of results with another
• Bootstrap (resampling) method
simple to apply
9
Accuracy – categorical verification
Estimated
yes
M
observations
H
F
CR
Observed
forecast
no
yes
H = hits
M = misses
no
F = false
alarms
CR = correct
rejections
Standard categorical verification scores
PC = (H + CR) / N
Bias = (F + H) / (M + H)
POD = H / (H + M)
POFD = F / (CR + F)
FAR = F / (H + F)
CSI = H / (H + M + F)
ETS = (H – Hrandom) / (H + M + F – Hrandom)
HSS = (H + CR – PCrandom) / (N – PCrandom)
HK = POD – POFD
OR = (H * CR) / (F * M)
proportion correct (accuracy)
frequency bias
probability of detection
probability of false detection
false alarm ratio
critical success index (threat score)
equitable threat score
Heidke skill score
Hanssen and Kuipers discriminant
odds ratio
10
Accuracy – continuous verification
Forecast
F
Observations
O
Domain
Standard continuous verification scores
(scores computed over entire domain)
bias = F  O
mean error
MAE = F  O
mean absolute error
1
( F  O )2

N
 ( F  F )( O  O )
RMSE =
r=
2
(
F

F
)

2
(
O

O
)

root mean square error
correlation coefficient
11
Accuracy – probabilistic verification
Standard probabilistic verification scores/methods
Reliability diagram
Relative operating characteristic (ROC)
1 N
BS 
( pi  oi )2

N i 1
BSS  1 
BS
BSreference
1 M
RPS 
( CDFfcst,m  CDFobs ,m )2

M  1 m 1
Brier score
Brier skill score
Ranked probability score 12
Strategy 1: Plot the performance of the
forecast system and the unskilled
reference on the same diagram
Skill scores measure the relative
improvement of the forecast over
the reference forecast:
skill 
scoreforecast  scorereference forecast
scoreperfect  scorereference forecast
Strategy 2: Plot the value of the skill score
0.6
> 0 mm
0.5
> 1 mm
> 5 mm
0.4
0.3
0.2
0.1
0
-0.1
30
60
90
120 150
Forecast (min)
180
____ Nowcast _ _ _ Extrapolation........ Gauge persistence
Skill w.r.t. gauge persis.
A forecast has skill if it is more
accurate than a reference forecast
(usually persistence, cell extrapolation, or random chance).
Hanssen & Kuipers score
Skill
0
-0.2
-0.4
-0.6
-0.8
-1
30
60
90
120
150
180
Forecast (min)
____ Nowcast _ _ _ Extrapolation
13
Practically perfect hindcast – upper bound on accuracy
Approach: If the forecaster had all of the observations in
advance, what would the "practically perfect" forecast look like?
• Apply a smoothing function to the observations to get probability
contours, choose an appropriate yes/no threshold
• Did the actual forecast look like the practically perfect forecast?
• How did the performance of the actual forecast compare to the
performance of the practically perfect forecast?
Convective outlook
was 75% of the way
to being "practically
perfect"
SPC convective outlook CSI = 0.34
Practically perfect hindcast CSI = 0.48
Kay and Brooks, 2000
14
"Double penalty"
Event predicted where it did not occur, no event predicted
where it did occur
Big problem for nowcasts and other high resolution forecasts
Ex: Two rain forecasts giving the same volume
fcst
obs
10
10
High resolution forecast
RMS ~ 4.7
POD=0, FAR=1, CSI=0
fcst
3
obs
10
Low resolution forecast
RMS ~ 2.7
POD~1, FAR~0.7, CSI~0.3
15
Value
A forecast has value if it helps a user make a better decision
Value scores measures the relative economic value of the forecast
over some reference forecast:
expenseforecast  expensereference forecast
value 
expenseperfect  expensereference forecast
obs
fcst
Small or rare events with high losses,
value maximized by over-prediction
Expense depends on the cost of
taking preventative action and the
loss incurred for a missed event
obs fcst
Events with high costs and displacement error
likely, value maximized by under-prediction
The most accurate forecast is not always the most valuable!
Baldwin and Kain, 2004
16
Exact match vs. "close enough"
Need we get a high resolution forecast exactly right?
Often "close" is still useful to a forecaster
YES
• High stakes situations (e.g. space
shuttle launch, hurricane landfall)
• Hydrological applications (e.g. flash
floods)
• Topographically influenced weather
(valley winds, orographic rain, etc.)
Standard verification methods
appropriate (POD, FAR, CSI,
bias, RMSE, correlation, etc.)
NO
• Guidance for forecasters
• Model validation (does it predict
what we expect it to predict?)
• Observations may not allow
standard verification of high
resolution forecasts
"Fuzzy" verification methods,
diagnostic methods
verify attributes of forecast
17
"Fuzzy" verification methods
• Large forecast and observed variability at high resolution
Sydney
Forecasters don't (shouldn't!) take a high resolution forecast
at face value – instead they interpret it in a probabilistic way.
• Fuzzy verification methods don't require an exact match between
forecasts and observations to get a good score
• Damrath, 2004
• Rezacova and Sokol, 2004 *
t-1
• Theis et al., 2005
t
• Roberts, 2004 *
t+1
Frequency
• Vary the size of the space / time neighborhood around a point
Forecast value
• Germann and Zawadski, 2004
• Also vary magnitude, other elements
• Atger, 2001
• Evaluate using categorical, continuous, probabilistic
scores / methods
* Giving a talk in this Symposium
18
Spatial multi-event contingency table
Verify using the Relative Operating
Characteristic (ROC)
• Measures how well the forecast can
separate events from non-events based
on some decision threshold
single threshold
Decision thresholds to vary:
• magnitude (ex: 1 mm h-1 to 20 mm h-1)
• distance from point of interest (ex:
within 10 km, .... , within 100 km)
• timing (ex: within 1 h, ... , within 12 h)
• anything else that may be important in
interpreting the forecast
Can apply to ensembles, and to compare
deterministic forecasts to ensemble
forecasts
Atger, 2001
ROC curve for varying rain threshold
ROC curve for ensemble forecast,
varying rain threshold
19
Object- and entity-based verification
• Consistent with human interpretation
• Provides diagnostic information on whole-system properties
•
•
•
•
Location
Amplitude
Size
Shape
NCAR
fcst
obs
Df
Cf
Bf
• Techniques
Af
Co
Bo
Do
Ao
• Contiguous Rain Area (CRA) verification (Ebert and McBride, 2000)
• NCAR object-oriented approach* (Brown et al., 2004)
• Cluster analysis (Marzban and Sandgathe, 2005)
• Composite method (Nachamkin, 2004)
8 clusters
identified in
x-y-p space
MM5
20
Contiguous Rain Area (CRA) verification
• Define entities using threshold (Contiguous Rain Areas)
• Horizontally translate the forecast until a pattern matching
criterion is met:
• minimum total squared error
• maximum correlation
• maximum overlap
Obs
Fcst
• The displacement is the vector difference between the original
and final locations of the forecast.
• Compare properties of matched entities
•
•
•
•
area
mean intensity
max intensity
shape, etc.
Ebert and McBride, 2000
21
Error decomposition methods
• Attempt to quantify the causes of the errors
• Some approaches:
• CRA verification (Ebert and McBride, 2000)
MSEtotal = MSEdisplacement + MSEvolume + MSEpattern
• Feature calibration and alignment (Nehrkorn et al., 2003)
E(x,y) = Ephase(x,y) + Elocal bias(x,y) + Eresidual(x,y)
• Acuity-fidelity approach (Marshall et al., 2004)
minimize cost function: J = Jdistance + Jtiming + Jintensity + Jmisses
from both perspectives of forecast (fidelity) and observations (acuity)
• Error separation (Ciach and Krajewski, 1999)
MSEforecast = MSEtrue + MSEreference
22
Scale separation methods
• Measure correspondence between forecast and observations at
a variety of spatial scales
• Some approaches:
• Multiscale statistical properties (Zepeda-Arce et al.,
2000; Harris et al., 2001)
• Scale recursive
estimation (Tustison et al.,
2003)
• Intensity-scale approach*
(Casati et al., 2004)
SATELLITE
l=0
MODEL
l=1
RADAR
l=2
RAIN
GAUGES
l=3
23
Summary
• Nowcasts and very short range forecasts present some unique
challenges for verification
• High impact weather
• High resolution forecasts
• Imperfect observations
• There is still a place for standard scores
•
•
•
•
•
Historical reasons
When highly accurate forecasts are required
Useful for monitoring improvement
Must use several metrics
Please quantify uncertainty, especially when intercomparing
forecast schemes
• Compare with unskilled forecast such as persistence
24
Summary (cont'd)
• Evolving concept of what makes a "good" forecast
• Recognizing value of "close enough"
• Probabilistic view of deterministic forecasts
• Exciting new developments of diagnostic methods to better
understand the nature and causes of forecast errors
• Object- and entity-based
• Error decomposition
• Scale separation
25
http://www.bom.gov.au/bmrc/wefor/staff/eee/verif/verif_web_page.html
26