Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Local Bayesian Model Averaging for the UW ProbCast Eric P. Grimit, Jeffrey Baars, Clifford F. Mass University of Washington, Atmospheric Sciences Patrick Tewson University of Washington, Applied Physics Laboratory Research supported by: Office of Naval Research Multi-Disciplinary University Research Initiative (MURI) Spring MURI Meeting; Seattle, WA 4 May 2006 10:50 AM Motivation “As high as 81! Hey, Eric! Those intervals are too wide! And 15% chance of precip? Hmm…” Spring MURI Meeting; Seattle, WA 4 May 2006 10:50 AM Summary from Last Fall Mean error climatology (MEC): Ensemble-mean + its error variance over some history. Good benchmark to evaluate competing calibration methods. Generally beats the raw ensemble, even though it is not a statedependent forecast of uncertainty. Local Bayesian model averaging (Local-BMA): Model forecast performance varies locally: BMA parameters should depend on grid point location. Train BMA using elevation, land-use, and proximity constraints. Can consistently beat MEC in tests with grid-based verification. Spring MURI Meeting; Seattle, WA 4 May 2006 10:50 AM Global-BMA Calibration and Sharpness calibration Probability integral transform (PIT) histograms an analog of verification rank histograms for continuous forecasts FIT MEC BMA [00 UTC Cycle; October 2002 – March 2004; 361 cases] sharpness MEC GlobalBMA Spring MURI Meeting; Seattle, WA 4 May 2006 10:50 AM Local-BMA Calibration and Sharpness calibration FIT MEC BMA [00 UTC Cycle; October 2002 – March 2004; 361 cases] sharpness MEC LocalBMA Probability integral transform (PIT) histograms an analog of verification rank histograms for continuous forecasts Spring MURI Meeting; Seattle, WA 4 May 2006 10:50 AM BMA Forecast Skill Comparison Global-BMA CRPS % improvement over MEC Local-BMA CRPS % improvement over MEC An Observation-Based Approach to Local-BMA Development and testing: Winter-Spring 2006 Expect it to drive the MURI “killer application” UW ProbCast. Several “tuning” parameters available, which can hopefully be optimized. Deploy it initially for MAXT2 and MINT2 forecasts. Application to mixed discrete-continuous quantities (e.g., QPF) and 2-D quantities (wind) will require further exploration. Spring MURI Meeting; Seattle, WA 4 May 2006 10:50 AM An Observation-Based Approach to Local-BMA Allow BMA parameters to vary by grid point. Use observations, remote if necessary, as training data. Follow the Baars et al. procedure for bias correction (optimized from the Mass-Wedam-Steed method) to also select the training data for Local-BMA. For each grid point, search for n (e.g. 8) nearby stations (e.g. within 864-km) at similar elevation (e.g. within 250-m) and having similar land-use. Land-use categories were concatenated into 9 categories (down from 24 in MM5). Figure shows methodology for Mass-Wedam-Steed settings. Spring MURI Meeting; Seattle, WA 4 May 2006 10:50 AM Maximum 2-m Temperature – Case Study ----Station: KHMS (Hanford, WA) Latitude, Longitude: 46.56, -119.60 South-north grid point: 52.289021 West-east grid point : 73.611740 obs: model#:1, forecast: model#:2, forecast: model#:3, forecast: model#:4, forecast: model#:5, forecast: model#:6, forecast: model#:7, forecast: model#:8, forecast: ENS-MEAN: ----- 71.00 F 64.24 F 67.12 F 62.01 F 60.36 F 62.30 F 61.59 F 64.80 F 66.88 F 63.67 F Spring MURI Meeting; Seattle, WA Global-BMA Mean ----Station: KHMS (Hanford, WA) Latitude, Longitude: 46.56, -119.60 South-north grid point: 52.289021 West-east grid point : 73.611740 obs: 71.00 F model#:1, forecast: 67.56 F model#:2, forecast: 69.57 F model#:3, forecast: 65.71 F model#:4, forecast: 64.62 F model#:5, forecast: 67.23 F model#:6, forecast: 66.28 F model#:7, forecast: 68.80 F model#:8, forecast: 69.23 F Global-BMA-MEAN: 65.11 F ----- 4 May 2006 10:50 AM Spring MURI Meeting; Seattle, WA Local-BMA Mean ----Station: KHMS (Hanford, WA) Latitude, Longitude: 46.56, -119.60 South-north grid point: 52.289021 West-east grid point : 73.611740 obs: model#:1, forecast: model#:2, forecast: model#:3, forecast: model#:4, forecast: model#:5, forecast: model#:6, forecast: model#:7, forecast: model#:8, forecast: Local-BMA-MEAN: ----- 71.00 F 67.56 F 69.57 F 65.71 F 64.62 F 67.23 F 66.28 F 68.80 F 69.23 F 68.84 F 4 May 2006 10:50 AM Spring MURI Meeting; Seattle, WA 4 May 2006 10:50 AM Bias-Corrected Ensemble Mean ----Station: KHMS (Hanford, WA) Latitude, Longitude: 46.56, -119.60 South-north grid point: 52.289021 West-east grid point : 73.611740 obs: model#:1, forecast: model#:2, forecast: model#:3, forecast: model#:4, forecast: model#:5, forecast: model#:6, forecast: model#:7, forecast: model#:8, forecast: BC-ENS-MEAN: ----- 71.00 F 67.56 F 69.57 F 65.71 F 64.62 F 67.23 F 66.28 F 68.80 F 69.23 F 67.38 F Spring MURI Meeting; Seattle, WA Global-BMA Sharpness ----Station: KHMS (Hanford, WA) Latitude, Longitude: 46.56, -119.60 South-north grid point: 52.289021 West-east grid point : 73.611740 obs: 71.00 F Global-BMA-95%: 72.55 F Global-BMA-MEAN: 65.11 F Global-BMA- 5%: 57.68 F ----- 4 May 2006 10:50 AM Spring MURI Meeting; Seattle, WA Local-BMA Sharpness ----Station: KHMS (Hanford, WA) Latitude, Longitude: 46.56, -119.60 South-north grid point: 52.289021 West-east grid point : 73.611740 obs: Local-BMA-95%: Local-BMA-MEAN: Local-BMA- 5%: ----- 71.00 F 73.64 F 68.84 F 63.36 F 4 May 2006 10:50 AM Spring MURI Meeting; Seattle, WA Local-MEC Sharpness ----Station: KHMS (Hanford, WA) Latitude, Longitude: 46.56, -119.60 South-north grid point: 52.289021 West-east grid point : 73.611740 obs: Local-MEC-95%: Local-MEC-MEAN: Local-MEC- 5%: ----- 71.00 F 72.15 F 67.38 F 62.61 F 4 May 2006 10:50 AM Spring MURI Meeting; Seattle, WA Calibration (all stations) 4 May 2006 10:50 AM Spring MURI Meeting; Seattle, WA Calibration (water only) 4 May 2006 10:50 AM Spring MURI Meeting; Seattle, WA Sharpness (all stations) 4 May 2006 10:50 AM Spring MURI Meeting; Seattle, WA Sharpness (water only) 4 May 2006 10:50 AM Spring MURI Meeting; Seattle, WA 4 May 2006 10:50 AM Minimum 2-m Temperature – Same Story (water only) (all stations) (calibration) (sharpness) Spring MURI Meeting; Seattle, WA 4 May 2006 10:50 AM Continuous Ranked Probability Scores (water only) (all stations) (MAXT2) (MINT2) Spring MURI Meeting; Seattle, WA 4 May 2006 10:50 AM Next Steps Go operational with Local-BMA for MAXT2 and MINT2. Code almost ready. Some issues remaining with “blank” grid points. Parameter optimization? Work on precip next (PoP & PQPF). Issues with small training samples and precip. What if all zeroes? Probably need to modify the search parameters. Distance to crest? Up-slope / down-slope? Depends on terrain gradient and wind! Wind (2-D vector). Established methods for wind speed and direction, separately. Use gamma and Von Mises mixture distributions, respectively. Need to build an EM-like algorithm or employ CRPS (energy score) minimization for 2-D wind forecasts. Work is being done on the CRPS (energy score) for 2-D variables. [statistics] QUESTIONS and DISCUSSION Spring MURI Meeting; Seattle, WA 4 May 2006 10:50 AM Spring MURI Meeting; Seattle, WA 4 May 2006 10:50 AM Spring MURI Meeting; Seattle, WA 4 May 2006 10:50 AM MEC Performance with Grid-Based Verification Comparison of *UWME 48-h 2-m temperature forecasts: Member-specific mean bias correction applied to both [14-day running mean] FIT = Gaussian fit to the raw forecast ensemble MEC = Gaussian fit to the ensemble-mean + the mean error climatology [00 UTC Cycle; October 2002 – March 2004; 361 cases] FIT MEC CRPS = continuous ranked probability score [Probabilistic analog of the mean absolute error (MAE) for scoring deterministic forecasts] Spring MURI Meeting; Seattle, WA 4 May 2006 10:50 AM Local-BMA Forecast Performance After several attempts to implement BMA with local or regional training data, EXCELLENT results were achieved: when the training data is selected from a neighborhood* of grid points with similar land-use type and elevation Example application to 48-h 2-m temperature forecasts uses only 14 training days. Dramatic improvements in CRPS nearly everywhere. *neighbors have same land use type and elevation difference < 200 m within a search radius of 3 grid points (60 km) MEC BMA Spring MURI Meeting; Seattle, WA 4 May 2006 10:50 AM An Advanced Calibration Method Bayesian Model Averaging (BMA) Summary BMA has several advantages over MEC: Member-specific mean-bias correction Member-specific BMA parameters weights BMA variance (not-member specific here, but can be) A time-varying uncertainty forecast. A way to keep multi-modality, if it is warranted. Maximizes information from short (2-4 week) training periods. Allows for different relative skill between members through the BMA weights (multi-model, multi-scheme physics). [c.f. Raftery et al. 2005, Mon. Wea. Rev.] Spring MURI Meeting; Seattle, WA 4 May 2006 10:50 AM Extending BMA to Non-Gaussian Variables For quantities such as wind speed and precipitation, distributions are not only non-Gaussian, but not purely continuous – there are point masses at zero. For probabilistic quantitative precipitation forecasts (PQPF): Model P(Y=0) with a logistic regression. Model P(Y>0) with a finite Gamma mixture distribution. Fit Gamma means as a linear regression of the cubed-root of observation on forecast and an indicator function for no precipitation. Fit Gamma variance parameters and BMA weights by the EM algorithm, with some modifications. [c.f. Sloughter et al. 200x, manuscript in preparation]