The WMO/WCRP/WWRP/COST sponsored International Workshop on Verification Methods was hosted by ECMWF from 29 January to 2 February 2007.
Description
The workshop included both a tutorial session and a program of talks on new developments in verification methodologies. The tutorial session provided a pedagogical introduction and review of verification techniques for both deterministic and probabilistic (including ensemble) forecasts with hands-on laboratory sessions, where participants used methodologies on real case studies. For these laboratory sessions, participants were invited to bring their own datasets and verification problems.
The second part of the workshop include keynote addresses focussing on new verification techniques and issues related to the practice of forecast verification, as well as contributed presentations on verification methodologies applied to a variety of forecasts (including forecasts of phenomena outside of atmospheric sciences, such as economics). Subjects will cover verification of ensemble/probability forecasts, extreme events, forecast value and user issues, and spatial forecasts.
Tutorials
Tutorial sessions ran from Monday 29 to Wednesday 31 January. The last session was combined with the Verification Workshop.
The participants of the tutorials worked on a project that was then presented on Friday 2 February during the morning session of the Verification Workshop.
The lecturers were Barbara Brown (NCAR, USA), Barbara Casati (MSC, Canada), Elizabeth Ebert (BOM, Australia), Simon Mason (IRI, USA), Pertti Nurmi (FMI, Finland), Matt Pocernich (NCAR, USA), Leonard Smith (LSE, UK), Laurence Wilson (MSC, Canada)
The following background readings were suggested:
- Verification
- “Forecast Verification : A Practitioner's Guide in Atmospheric Science" by Ian T. Jolliffe (Editor), David B. Stephenson (Editor), Wiley
- “Statistical Methods in the Atmospheric Sciences" (Chapter 7), by D.S.Wilks, 2006, Elsevier
- Joint Working Group on Verification
- R package
- Modules on Forecast Verification (EUMETCAL)
- EUMETCAL Training Modules
Programme
Presentations
| Session 1: Forecast Value | |
|---|---|
|
Ensemble forecast: their statistics, value and application Lenny Smith |
 |
| Session 2: Ensemble Verification | |
|---|---|
|
Ensemble forecast verification - Recent work, new ideas and issues Laurence Wilson |
|
|
Probabilistic verification of SRNWP-PEPS Michael Denhard |
|
|
Estimating the effect of ensemble size on the verification measures for binary-event forecasts Christopher Ferro |
|
|
Verifying the relationship between ensemble spread and skill Thomas Hopson |
|
|
Using time-lag ensemble techniques to assess the behaviour of high-resolution precipitation forecasts Marion Mittermaier |
|
|
Using Bayesian model averaging to calibrate short range forecasts form a multi-model ensemble Daniel Santos-Munoz |
|
|
A multi-model multi-analysis limited area ensemble: calibration issues Piero Chessa |
|
| Session 3: Spatial Verification and Scale Separation | |
|---|---|
|
Verification techniques for spatial forecasts Barbara Casati |
|
|
Adaptation of the intensity-scale verification technique for precipitation over Hungary Gabriella Csima |
|
|
Assessing high resolution forecasts using fuzzy verification methods Elizabeth Ebert |
|
|
Probabilistic approach in comparative verification of high resolution models Kees Kok |
|
|
The use of a method for determining the variation of precipitation forecast skill with spatial scale Nigel Roberts |
|
|
Short, medium range and seasonal forecasts verification methods for the AMMA-EU project Andre Kamga |
|
|
Diagnostic evaluation of precipitation and convection forecast at multiple spatial scales Barbara Brown |
|
|
Developments in object-based verification: distribution of object attributes and incorporation of the time dimension Chris Davis |
|
|
Verification via optical flow Caren Marzban |
|
|
SAL - a novel error measure for the high resolution precipitation forecasts Heini Wernli |
|
|
An intercomparison of spatial forecast verification techniques Eric Gilleland |
|
| Session 4: User-oriented Verification | |
|---|---|
|
Verification of extreme/high impact weather Christopher Ferro and David Stephenson |
|
|
Different types of verification results required by different users Ulrich Damrath |
|
|
Could a perfect model ever satisfy the forecaster? A grid-box versus point forecast Ervin Zsoter |
|
| Session 5: Interpretation of Verification Measures | |
|---|---|
|
Do high skill scores mean good forecast? Simon Mason |
|
|
The effect of observation errors on verification Neill Bowler |
|
|
The concept of 'Deterministic limit' Tim Hewson |
|
|
Quantile based point-to-point validations Johannes Jenkner |
|
|
Playing the score - exploring beyond the hedge Ian Jolliffe |
|
|
Two extra components in the Brier score decomposition David Stephenson (Presented by Caio Coelho) |
|
|
Another look at proper scoring rules Laurence Wilson |
|
|
The discrete Brier and ranked probability skill scores Andreas Weigel |
|
|
Comparing peaches and apples - on the accuracy of weather warning issues by forecasters and the accuracy of the model guidance Martin Goeber |
|
| Session 6: Verification Packages | |
|---|---|
|
PEIXE: an application for verification of forecast time series Breogan Gomez |
|
|
Development of a state-of-the-art automated verification package Lacey Holland |
|
| Session 7: Verification projects | |
|---|---|
|
Verification of PoP forecasts for Hong Kong A Maidens, A Celozzi, Y-F Tong, C Gebhardt and P Zacharov |
|
|
Temperature verification C Cretu, M Benko, A Sarr, H Persson and G T Diro |
|
|
Skill of precipitation forecasts at Takle, Norway G Hauge, M Patarcic, T M Madsen, E Hansen, H-S Jung and Y-Q Wang |
|
|
Wind speed verification for Quebec City, Canada M Moreau, M Amodei, K Salonen, T Phan and C Crosbie |
|
|
Climatological analysis of daily rainfall amounts over Addis Ababa D Korecha, John Gitutu, J Ramasamy and A Tladi |
|
|
Precipitation forecasts in Argentina V Rudak, O Kryvobok, J Garcia-Moya, H Myllyniemi and M Suaya |
|
|
The intensity scale verification method with missing values J Jenkner and F Liang |
|
| SESSION 8: Verification Applications | |
|---|---|
|
Probabilistic Verification at high-resolution of the COSMO-LEPS limited area ensemble Chiara Marsigli |
|
|
Performance of the INM short range multi-model ensemble using high-resolution precipitation observations Carlos Santos |
|
|
Verifying Hydrological forecasts in the US National Weather Service Julie Demargne |
|
|
Seasonal forecasting of the Ethiopian rainfall using SST Gulilat Tefera Diro |
|
|
A comparative verification of precipitation forecasts for selected NWP models over Kenya John Gitutu |
|
|
Bias estimation of Doppler radar radial winds Kirsti Salonen |
|
|
Verification of the COSMO LM 7KM- version at Meteo-Swiss and new developments for the 2.2KM-version with a special emphasis on precipitation Francis Schubiger |
|
|
Satellite-based real-time monitoring and verification of model forecasted clouds Chrisoph Zingerie |
|
|
Satellite based analysis of the capability of a climate model to reproduce short time scale cloud variability over the Mediterranean area. Preliminary results: summer-fall 2000 G L Liberti |
|
Tutorial sessions
|
Introduction to R Matt Pocernich |
|
|
Basic Verification Concepts Barbara Brown |
|
|
Verification of Continuous Predictands Barabara Casati |
|
|
Verification of Categorical Predictands Pertti Nurmi |
|
|
Verification of Probabilistic Predictands and Ensemble Forecasts Beth Ebert |
|
|
Confidence Interval and Hypothesis Tests Simon Mason |
|
Posters
|
Verification at the Croatian Meteorological and Hydrological Service Lovro Kalin |
|
|
TAF and TREND verification in the Austro Control using time windows Guenter Mahringer |
|
|
Displacement errors of quantitative precipitation forecasts over the Calabria region Nazario Trataglione |
|
|
The impact of range-adjusted ground-based radar observations on CRA rainfall verification Christophe Accadia |
|
|
Conditional verification for high resolution LAM Angela Celozzi |
|
|
NWP-model verification with VERA over complex terrain M Dorninger |
|
|
Assessing the value of the local heatwave forecasts in Lombardia by surveying and monitoring end users Gianpaolo Minardi |
|
|
Verification ideas arising from the University of South Carolina forecasting competition Christopher Bell |
- |
|
Uncertainties in the high resolution model verification - the case of ETA model performance in Argentina Martina Suaya |
|
|
Statistical downscaling of the warm season precipitation in the core North American monsoon region Kazungu Maitaria |
- |
|
Verification of an hybrid short range ensemble prediction system Alfons Callado |
|
|
A two-week temperature probability distribution function at the South African Weather Service Alfred Tladi |
- |
|
Mediterranean ocean forecasting system: forecast analysis and assessment through skill scores Marina Tonani |
|
|
Assessing simulation of Western Africa present time using the regional climate model RegCM3 Adboulaye Sarr |
|
|
Verification of sand and dust storm forecasts in NorthEast Asia from CUACE-Dust model Yan-Quian Wang |
- |
|
Investigating and verifying the skill of ECMWF deterministic weather forecasts with respect to the case of flooding events of summer 2006 over Ethiopia Diriba Korecha |
|
|
Verification methods fore precipitation and temperature Hakan Persson |
- |
|
Comparison of precipitation forecasts by two operational NWP models: the global ARPEGE and the smaller mesh LAM ALADIN Marielle Amodei |
|
|
The verification of high resolution forecasts of the operational JMA mesoscale model Tomonori Segawa |
|
|
Results on precipitation verification over Italy Elena Oberto |
|
|
Performance and application of the ensemble prediction system (EPS) in Hong Kong Yu-Fai Tong |
|
|
Probabilistic evaluations of a cloud system model Peter Henderson and Robert Pincus |
- |
|
Probabilistic forecasts based on the Hirlam IMN deterministic model Juan Simarro |
|
|
Testing geometric bred vectors with a mesoscale short range ensemble prediction system over the Western Mediterranean Alberto Martin |
|
|
Multi-model ensemble for short range predictions Jose Garcia Moya |
|
|
Performance of the MOGREPS short range ensemble prediction system K Mylne |
- |