Download Assessing forecast uncertainty from synoptic to sub

Celeste Saulo and Juan Ruiz
CIMA (CONICET/UBA) – DCAO (FCEN –UBA)
Motivation and general context




Many meteorological services run operational
ensemble prediction systems (EPS), which provide
estimates of the uncertainty of the forecast.
Many of these outputs are readily available to the
scientific community through, e.g. TIGGE
(THORPEX Interactive Grand Global Ensemble).
Obtaining useful (valuable) information from EPS
requires statistical post-processing and specific
research depending on the variable/problem/region.
There is growing interest in obtaining useful
information from EPS on time scales between 2
weeks and 2 months.
Motivation and general context
Active research is being pursued in numerous
places on the definition of initial ensembles,
multimodel (or stochastic physics) as well as on
the evaluation of ensemble predictions.
 During the first half of THORPEX it was realized
that model error diagnosis is one area where
universities and research institutions can make
substantial contributions to the further
development of models (and hence forecast
skill), thereby supporting the relatively small
community of model developers.

THORPEX = The Observing System Research and Predictability Experiment
Potential areas of research under
UMI-IFAECI
Predictability studies
 Ensemble generation (including data
assimilation)
 Probabilistic forecasts
 Verification strategies

Related ongoing studies

How sensitive are probabilistic precipitation
forecasts to the choice of calibration
algorithms and the ensemble generation
method?
 Part I: Sensitivity to calibration methods (Ruiz
and Saulo, Meteorol. Appl., 2011)
 Part II: sensitivity to ensemble generation
method (Ruiz, Saulo and Kalnay, Meteorol. Appl.
2011)

Three different ensemble generation strategies,
using WRF regional model as the basis:
 Breeding (11 members)
 Multi-model (11 members)
 Pragmatic= spatially shifted ((2*m + 1)2 members, e.g.,
121)

In order to correct the effect of the ensemble systematic
errors, several techniques have been developed, all of
them based in the study of the relationship between
error and forecasted value and in the development of
statistical models to compute a calibrated probability
given the forecasts of the ensemble members
 A logistic regression is used to represent h(y>0|f) and a
GAMMA function is used to represent h(y>tr|f,y>0)
 BMA → weighted + calibrated probability for each
member
 GAMMA-ENS →all weights are equal + calibrated
probability for each member
 GAMMA→ no weights + calibration applied to the
ensemble mean
 WMEAN →weighted ensemble mean and then
calibration is applied
Weights associated to each member of the spatially
shifted ensemble as a function of the corresponding shift
in the south–north (y axis) and the west–east (x axis)
directions. Negative shift values indicate southward and
westward shifts respectively.
Continuous ranked
probability score (CRPS)
GAMMA calibration
has been adopted
The computation of a weighted ensemble mean can lead to
moderate better results; however the best choice for a weight
computation algorithm is still an open question. The PQPF
derived from the un-weighted ensemble mean produces, if not
the best results, almost as good results as any other approach.
Shifted
MM
Combined
Breeding
24 hours forecast
48 hours forecast
Shifted-MM
Shifted combined
shifted multimodel 1331 members
shifted breeding 1331 members
shifted combined 2541 members
ShiftedBreeding

The spatially shifted ensemble proves to
be quite competitive at short forecast
ranges,
Precipitation uncertainty at
these ranges is mostly related
with the location of rain areas

yet its skill drops rapidly with increasing
lead times
uncertainties associated with the
existence, or intensity of pp, tend to
become more important with
increasing lead times.

multimodel ensemble (physics)
outperformed the breeding ensemble
(I&BC). Still, the improvement combining
both is modest
most of the PQPF limitations during
summer arise from errors in model
physics rather than problems in the
initial and/or boundary conditions

Among the alternatives that have been
evaluated, the most important improvement
has been obtained with the combination of the
multimodel ensemble approach (and/or the
combined approach) and the spatial shift
technique even at 48-hours lead time. This
approach is particularly interesting and
promising for implementing high resolution
ensembles in small operational or research
centers for which computational costs largely
restrict ensemble size.
Ensemble Forecast Object
Oriented Verification Method




Work in progress Juan Ruiz (postdoc at LMD) and
Olivier Talagrand
The method has been designed to be applied to
the 500 hPa field, however it can be easily
extended to other fields as well (and probably
other “objects” i.e. jet streak position, low level jet
maximum possition, etc).
It is based in the identification of local minima and
the system associated with each local minima.
As in 500 hPa, usually low pressure systems
appear in the form of troughs rather than in the
form of closed systems, the geopotential height
anomaly is used instead of the full 500 hPa field.
Cyclone trajectories at 500 hPa, for a particular
day derived from the NCEP ensemble system
Questions for future research





How much information can be obtained from the ensemble
spread about the forecast skill? Are there specific scores to
quantify this relationship in terms that it becomes useful for
particular applications?
Which is the most convenient way to combine different
ensemble members? Is it necessary to take into account
the different skill of each member? (i.e. Bayesian model
averaging trying different weights against simpler
techniques like logistic regression for precip)
Which kind of information/type of scores could be used to
provide valuable information about weather states with
more than two weeks in advance?
How can we use model error statistics to understand which
processes are strongly affecting forecast quality so that key
problems can be isolated and models improved?
Which methodologies should we apply to forecast
probability of extreme events?