We continually monitor the accuracy of our operational forecasts. This evaluation provides essential feedback to both users and model developers on the quality of the forecasting system.
Operational forecast evaluation
The overall performance of the operational medium-range forecasts is summarised using a set of headline scores which highlight different aspects of forecast skill.
These headline scores, together with additional verification of medium-range forecast performance are regularly updated on the ECMWF web site.
Evaluation is also carried out for the monthly and seasonal forecasts and results are available on the web.
We publish a detailed report on the performance of our forecasts every year as an ECMWF Technical Memorandum. This document presents recent verification statistics and evaluations of ECMWF forecasts (including weather, waves and severe weather events) along with information about changes to the data assimilation and forecasting system. The performance of the monthly and seasonal forecasts is also included.
The 2014-2015 report is15275-evaluation-ecmwf-forecasts-including-2014-2015-upgrades.pdf
In line with ECMWF’s Strategy 2011–2020, we have defined a set of two primary and four supplementary headline scores to evaluate long-term trends in forecast performance. The aim of this set of scores is to assess performance for various forecast lead times for surface weather parameters (such as precipitation and wind gusts) as well as for the traditional upper-air fields. Four of the headline scores (two primary and two secondary) are expressed in terms of the lead time at which the score reaches a specific threshold value. The thresholds have been chosen to target the verification on the relevant forecast range for each measure of skill.
The two primary scores are:
- forecast lead-time at which the anomaly correlation of the HRES 500 hPa geopotential reaches 80% for the extra-tropical northern hemisphere; verification against analyses
- forecast lead-time at which the continuous ranked probability skill score (CRPSS) for ENS probabilistic forecasts of 850 hPa temperature reaches 25% for the extra-tropical northern hemisphere; verification against analyses
The supplementary scores are:
- forecast lead-time at which the SEEPS score for HRES forecasts of 24-hour total precipitation reaches 45% for the extra-tropics (northern and southern hemispheres); verification against station observations
- forecast lead-time at which the continuous ranked probability skill score (CRPSS) for ENS probabilistic forecasts of 24-hour total precipitation reaches 10% for the extra-tropics (northern and southern hemispheres); verification against station observations
- HRES tropical cyclone position error at forecast day 3
- Extreme Forecast Index (EFI) skill of 10m wind speed at forecast day 4, evaluated using the Relative Operating Characteristic (ROC); verification against station observations in Europe
The primary scores and the supplementary scores for precipitation are regularly updated and are displayed on our web site. The two remaining scores (for severe weather) are updated annually and presented in the detailed report on the performance of the ECMWF. The 2012–2013 report is ECMWF Tech. Memo. No. 710 on ‘Verification statistics and evaluation of ECMWF forecasts in 2012–2013’.
Comparison with other centres
The World Meteorological Organisation (WMO) has defined procedures for producing and exchanging standard sets of verification scores. This allows us to compare the quality of our forecasts with those from other centres using a consistent evaluation framework.
The quality of our medium-range forecasts compared to those from other global centres can be seen on the web sites of the designated WMO Lead Centres:
- WMO Lead Centre for Deterministic Forecast Verification
- WMO Lead Centre for Verification of Ensemble Prediction System
The long-range forecast skill is measured using a set of verification scores that conform to an agreed WMO standard and is based on the re-forecast data set (30 years). Because this verification sample size is rather limited, skill measures are provided together with statistical significance levels. The skill depends strongly on the season, so forecasts for different starting months are evaluated separately.
The verification for the long-range forecasts is shown along with the relevant forecast charts. See it by selecting from the "Forecast type and skill measures" drop-down menu above each chart. Note that verification is not yet available on this web site for the climagrams or tropical storms.
Feedback from Member and Co-operating States
Each year ECMWF invites its Member and Co-operating States to provide feedback about their use of ECMWF forecasts. The national reports they provide include sections on verification. Follow the links below to see reports from recent years.