3rd International Workshop on Verification Methods

The WMO/WCRP/WWRP/COST sponsored International Workshop on Verification Methods was hosted by ECMWF from 29 January to 2 February 2007.

Description

The workshop included both a tutorial session and a program of talks on new developments in verification methodologies. The tutorial session provided a pedagogical introduction and review of verification techniques for both deterministic and probabilistic (including ensemble) forecasts with hands-on laboratory sessions, where participants used methodologies on real case studies. For these laboratory sessions, participants were invited to bring their own datasets and verification problems.

The second part of the workshop include keynote addresses focussing on new verification techniques and issues related to the practice of forecast verification, as well as contributed presentations on verification methodologies applied to a variety of forecasts (including forecasts of phenomena outside of atmospheric sciences, such as economics). Subjects will cover verification of ensemble/probability forecasts, extreme events, forecast value and user issues, and spatial forecasts.

Tutorials

Tutorial sessions ran from Monday 29 to Wednesday 31 January. The last session was combined with the Verification Workshop.

The participants of the tutorials worked on a project that was then presented on Friday 2 February during the morning session of the Verification Workshop.

The lecturers were Barbara Brown (NCAR, USA), Barbara Casati (MSC, Canada), Elizabeth Ebert (BOM, Australia), Simon Mason (IRI, USA), Pertti Nurmi (FMI, Finland), Matt Pocernich (NCAR, USA), Leonard Smith (LSE, UK), Laurence Wilson (MSC, Canada)

The following background readings were suggested:

  • Verification
    • “Forecast Verification : A Practitioner's Guide in Atmospheric Science" by Ian T. Jolliffe (Editor), David B. Stephenson (Editor), Wiley
    • “Statistical Methods in the Atmospheric Sciences" (Chapter 7), by D.S.Wilks, 2006, Elsevier
    • Joint Working Group on Verification
  • R package
  • Modules on Forecast Verification (EUMETCAL)
    • EUMETCAL Training Modules

Programme

PDF iconProgramme (Wednesday)

PDF iconProgramme (Thursday)

PDF iconProgramme (Friday)

Presentations

Session 1: Forecast Value  

Ensemble forecast: their statistics, value and application

Lenny Smith

PDF iconPDF icon
Session 2: Ensemble Verification  

Ensemble forecast verification - Recent work, new ideas and issues

Laurence Wilson

PDF icon

Probabilistic verification of SRNWP-PEPS

Michael Denhard

PDF icon

Estimating the effect of ensemble size on the verification measures for binary-event forecasts

Christopher Ferro

PDF icon

Verifying the relationship between ensemble spread and skill

Thomas Hopson

PDF icon

Using time-lag ensemble techniques to assess the behaviour of high-resolution precipitation forecasts

Marion Mittermaier

PDF icon

Using Bayesian model averaging to calibrate short range forecasts form a multi-model ensemble

Daniel Santos-Munoz

PDF icon

A multi-model multi-analysis limited area ensemble: calibration issues

Piero Chessa

PDF icon
Session 3: Spatial Verification and Scale Separation  

Verification techniques for spatial forecasts

Barbara Casati

PDF icon

Adaptation of the intensity-scale verification technique for precipitation over Hungary

Gabriella Csima

PDF icon

Assessing high resolution forecasts using fuzzy verification methods

Elizabeth Ebert

PDF icon

Probabilistic approach in comparative verification of high resolution models

Kees Kok

PDF icon

The use of a method for determining the variation of precipitation forecast skill with spatial scale

Nigel Roberts

PDF icon

Short, medium range and seasonal forecasts verification methods for the AMMA-EU project

Andre Kamga

PDF icon

Diagnostic evaluation of precipitation and convection forecast at multiple spatial scales

Barbara Brown

PDF icon

Developments in object-based verification: distribution of object attributes and incorporation of the time dimension

Chris Davis

PDF icon

Verification via optical flow

Caren Marzban

PDF icon

SAL - a novel error measure for the high resolution precipitation forecasts

Heini Wernli

PDF icon

An intercomparison of spatial forecast verification techniques

Eric Gilleland

PDF icon
Session 4: User-oriented Verification  

Verification of extreme/high impact weather

Christopher Ferro and David Stephenson

PDF iconPDF icon

Different types of verification results required by different users

Ulrich Damrath

PDF icon

Could a perfect model ever satisfy the forecaster? A grid-box versus point forecast

Ervin Zsoter

PDF icon
Session 5: Interpretation of Verification Measures  

Do high skill scores mean good forecast?

Simon Mason

PDF icon

The effect of observation errors on verification

Neill Bowler

PDF icon

The concept of 'Deterministic limit'

Tim Hewson

PDF icon

Quantile based point-to-point validations

Johannes Jenkner

PDF icon

Playing the score - exploring beyond the hedge

Ian Jolliffe

PDF icon

Two extra components in the Brier score decomposition

David Stephenson (Presented by Caio Coelho)

PDF icon

Another look at proper scoring rules

Laurence Wilson

PDF icon

The discrete Brier and ranked probability skill scores

Andreas Weigel

PDF icon

Comparing peaches and apples - on the accuracy of weather warning issues by forecasters and the accuracy of the model guidance

Martin Goeber

PDF icon
Session 6: Verification Packages  

PEIXE: an application for verification of forecast time series

Breogan Gomez

PDF icon

Development of a state-of-the-art automated verification package

Lacey Holland

PDF icon
Session 7: Verification projects  

Verification of PoP forecasts for Hong Kong

A Maidens, A Celozzi, Y-F Tong, C Gebhardt and P Zacharov

PDF icon

Temperature verification

C Cretu, M Benko, A Sarr, H Persson and G T Diro

PDF icon

Skill of precipitation forecasts at Takle, Norway

G Hauge, M Patarcic, T M Madsen, E Hansen, H-S Jung and Y-Q Wang

PDF icon

Wind speed verification for Quebec City, Canada

M Moreau, M Amodei, K Salonen, T Phan and C Crosbie

PDF icon

Climatological analysis of daily rainfall amounts over Addis Ababa

D Korecha, John Gitutu, J Ramasamy and A Tladi

PDF icon

Precipitation forecasts in Argentina

V Rudak, O Kryvobok, J Garcia-Moya, H Myllyniemi and M Suaya

PDF icon

The intensity scale verification method with missing values

J Jenkner and F Liang

PDF icon
SESSION 8: Verification Applications  

Probabilistic Verification at high-resolution of the COSMO-LEPS limited area ensemble

Chiara Marsigli

PDF icon

Performance of the INM short range multi-model ensemble using high-resolution precipitation observations

Carlos Santos

PDF icon

Verifying Hydrological forecasts in the US National Weather Service

Julie Demargne

PDF icon

Seasonal forecasting of the Ethiopian rainfall using SST

Gulilat Tefera Diro

PDF icon

A comparative verification of precipitation forecasts for selected NWP models over Kenya

John Gitutu

PDF icon

Bias estimation of Doppler radar radial winds

Kirsti Salonen

PDF icon

Verification of the COSMO LM 7KM- version at Meteo-Swiss and new developments for the 2.2KM-version with a special emphasis on precipitation

Francis Schubiger

PDF icon

Satellite-based real-time monitoring and verification of model forecasted clouds

Chrisoph Zingerie

PDF icon

Satellite based analysis of the capability of a climate model to reproduce short time scale cloud variability over the Mediterranean area. Preliminary results: summer-fall 2000

G L Liberti

PDF icon

Tutorial sessions

Introduction to R

Matt Pocernich

PDF icon

Basic Verification Concepts

Barbara Brown

PDF icon

Verification of Continuous Predictands

Barabara Casati

PDF icon

Verification of Categorical Predictands

Pertti Nurmi

PDF icon

Verification of Probabilistic Predictands and Ensemble Forecasts

Beth Ebert

PDF icon

Confidence Interval and Hypothesis Tests

Simon Mason

PDF icon

Posters

Verification at the Croatian Meteorological and Hydrological Service

Lovro Kalin

PDF icon

TAF and TREND verification in the Austro Control using time windows

Guenter Mahringer

PDF icon

Displacement errors of quantitative precipitation forecasts over the Calabria region

Nazario Trataglione

PDF icon

The impact of range-adjusted ground-based radar observations on CRA rainfall verification

Christophe Accadia

PDF icon

Conditional verification for high resolution LAM

Angela Celozzi

PDF icon

NWP-model verification with VERA over complex terrain

M Dorninger

PDF icon

Assessing the value of the local heatwave forecasts in Lombardia by surveying and monitoring end users

Gianpaolo Minardi

PDF icon

Verification ideas arising from the University of South Carolina forecasting competition

Christopher Bell

-

Uncertainties in the high resolution model verification - the case of ETA model performance in Argentina

Martina Suaya

PDF icon

Statistical downscaling of the warm season precipitation in the core North American monsoon region

Kazungu Maitaria

-

Verification of an hybrid short range ensemble prediction system

Alfons Callado

PDF icon

A two-week temperature probability distribution function at the South African Weather Service

Alfred Tladi

-

Mediterranean ocean forecasting system: forecast analysis and assessment through skill scores

Marina Tonani

PDF icon

Assessing simulation of Western Africa present time using the regional climate model RegCM3

Adboulaye Sarr

PDF icon

Verification of sand and dust storm forecasts in NorthEast Asia from CUACE-Dust model

Yan-Quian Wang

-

Investigating and verifying the skill of ECMWF deterministic weather forecasts with respect to the case of flooding events of summer 2006 over Ethiopia

Diriba Korecha

PDF icon

Verification methods fore precipitation and temperature

Hakan Persson

-

Comparison of precipitation forecasts by two operational NWP models: the global ARPEGE and the smaller mesh LAM ALADIN

Marielle Amodei

PDF icon

The verification of high resolution forecasts of the operational JMA mesoscale model

Tomonori Segawa

PDF icon

Results on precipitation verification over Italy

Elena Oberto

PDF icon

Performance and application of the ensemble prediction system (EPS) in Hong Kong

Yu-Fai Tong

PDF icon

Probabilistic evaluations of a cloud system model

Peter Henderson and Robert Pincus

-

Probabilistic forecasts based on the Hirlam IMN deterministic model

Juan Simarro

PDF icon

Testing geometric bred vectors with a mesoscale short range ensemble prediction system over the Western Mediterranean

Alberto Martin

PDF icon

Multi-model ensemble for short range predictions

Jose Garcia Moya

PDF icon

Performance of the MOGREPS short range ensemble prediction system

K Mylne

-

 

Local information


You can also see an archive of past workshops

View full calendar of events