ECMWF maintains a comprehensive range of verification statistics to evaluate the accuracy of its forecasts. Each year, a summary of verification results is presented to ECMWF’s Technical Advisory Committee (TAC). Their views about the performance of the operational forecasting system in 2019 are given in the box.
The overall performance of the operational forecasts is summarised using a set of headline scores endorsed by the TAC, which highlight different aspects of forecast skill. Two ensemble headline scores are shown here. Upper-air performance of the ensemble forecast (ENS) is monitored through the continuous ranked probability score (CRPS) for temperature at 850 hPa over the northern hemisphere extratropics. Forecast skill during summer 2019 was higher than in previous summer seasons, and comparison with re-forecast skill based on the ERA5 reanalysis shows that this is partly due to the latest upgrade of the Integrated Forecasting System (IFS Cycle 46r1 on 11 June 2019). This upgrade has brought improvements for both the ENS and the high-resolution forecast (HRES) across a range of parameters and atmospheric levels, including surface weather parameters. For example, the second headline score shown here, which monitors forecast skill for strong winds in terms of the Extreme Forecast Index, reached its highest ever value in 2019. There was also a slight increase in forecast skill for 2-metre temperature, and a gain in precipitation forecast skill.
The position error for forecasts of tropical cyclones increased compared to the previous year, but this was due to atmospheric variability, as ERA5 shows a very similar increase. HRES tropical cyclone intensity errors (as measured by central pressure) have reached their smallest values so far.
With regard to ocean waves, ECMWF has regained its lead compared to other global wave forecasting systems for forecasts of significant wave height. Wave parameters (significant wave height and mean wave period) in the HRES are improved in Cycle 46r1 by 5-10% due to a major upgrade in the ocean wave model. Increased wave activity leads to some degradation in wave height at longer lead times in the ENS. For peak period, ECMWF generally ranks second.
There has been no significant change in the headline score which monitors extended-range forecast performance, focusing on the ENS probabilistic skill for weekly mean 2-metre temperature forecasts in the northern extratropics in week 3 of the forecast (days 15–21). While forecast skill in week 2 shows a weak upward trend, no such signal can be detected for week 3.
On the seasonal timescale, the main driver of predictability in 2019 was an exceptionally strong positive phase of the Indian Ocean Dipole, which peaked in October, and which was predicted by ECMWF’s long- range forecasts 6 months ahead. This led to an unusually active tropical cyclone season in the Indian Ocean and record warm anomalies in Australia. There was no strong forcing from the El Niño–Southern Oscillation, which remained in a slightly positive phase. Heatwave conditions in Europe in summer 2019 were qualitatively predicted 1–2 months ahead, but their magnitude was captured only at shorter lead times (week 2).
Each summer, Member and Co-operating States report on the application and verification of ECMWF’s forecast products for the previous year. Many compare HRES with limited-area models (LAMs), and so usually centre on shorter ranges (up to about 60 hours). A very common finding, for almost every weather parameter, was that HRES forecast biases have a diurnal cycle, and annual cycles are also often present.
Large HRES forecast errors for 2-metre temperatures, in very hot or very cold situations, are a concern for some Member States, notably in Scandinavia, where very cold winter nights are not nearly cold enough in HRES forecasts. ECMWF continues to work on this model issue. Meanwhile, using the insightful approach of conditional verification (a growth area at ECMWF), Finland has shown that 2-metre temperature errors in winter tend to be lowest when the skin temperature (ordinarily of a snow surface) is about 0°C. This is because energy exchanges then involve the latent heat of fusion more than temperature change.
ENS-related verification results were limited, but Germany reported a multi-parameter, multi-pressure level comparison between the ICON and ECMWF ensembles, which they use as a verification benchmark for their system. Results for the stratosphere highlighted the known IFS bias which will be address in IFS Cycle 47r1 later this year.
The pivotal importance of correctly predicting severe events was reiterated in many reports. For example, extreme Mediterranean cyclones, around 28 September and 29 October 2018, were mentioned by several countries. These cases also appear in ECMWF’s severe event catalogue.
The complete set of annual results is available in two ECMWF Technical Memorandums, No. 853 on ‘Evaluation of ECMWF forecasts, including the 2019 upgrade’ and No. 860 on ‘Use and Verification of ECMWF products in Member and Co-operating States (2019)’. Both are downloadable from https://www.ecmwf.int/en/publications/technical-memoranda.
The following are other sources of information about verification and forecasting system changes.
- Verification pages: https://www.ecmwf.int/en/forecasts/charts
- Inter-comparisons of global model forecast skill: https://apps.ecmwf.int/wmolcdnv/
- A list of ‘Known IFS Forecasting Issues’: https://confluence.ecmwf.int/display/FCST/Known+IFS+forecasting+issues
- All IFS forecasting system cycle changes since 1985: https://www.ecmwf.int/en/forecasts/documentation-and-support/changes-ecmwf-model
Assessment of ECMWF’s Technical Advisory Committee, 10–11 October 2019
With regard to its overall view of the performance of ECMWF’s operational forecasting system, the Committee:
- noted that ECMWF headline scores continue to show high and improving skill, especially in the light of the introduction of 46r1; noted that the lead over other centres has been maintained and in some instances the lead in specific elements, e.g. significant wave height, has been regained, and noted that scores for some elements were the highest ever;
- noted that precipitation verification scores have started to improve again following a recent drop due to natural variability in predictability;
- welcomed improvements seen in the newly introduced score for large ENS 10 m wind errors;
- noted that ECMWF SEAS5 highlighted a warm anomaly for summer 2019 over Europe and ECMWF ENS highlighted heat waves at shorter lead times; however, at the intermediate lead time of 3-4 weeks, the forecast performed less well;
- recognised that the warm anomaly winter 2018/19 over Europe was not forecast at the seasonal time range;
- welcomed continued improvement to the forecasting of tropical cyclone tracks and welcomed the proposed investigation into tropical cyclone negative speed bias and intensity next year; investigation on the extratropical transition would also be welcome;
- acknowledged deficiencies in forecasting elevated convection and modelling of snow, and welcomed ongoing and proposed investigations into these topics;
- welcomed the development of ecCharts2 and the improvements in speed;
- welcomed proposed improvements to ensemble vertical and horizontal resolution at 48r1 and encouraged consideration of changing the extended-range ENS to run daily;
- appreciated the very good support ECMWF provided to Member and Co-operating States over the last year, in particular for high-impact weather events, and welcomed specific new products such as the point rainfall product, early warning for cold spells, extended EFI and integrated water vapour EFI;
- appreciated ECMWF’s invitation to Member and Co-operating States to identify additional products and services for use by forecasters, and appreciated ECMWF’s management of these requests with the URMS to assess priorities;
- welcomed the testing of OPERA data in both verification and data assimilation;
- appreciated the consistent and reliable delivery of ECMWF products; n) appreciated the earlier delivery of ENS and extended ENS output and recognised this may change with 48r1 and welcomed ECMWF’s optional offer to deliver products as soon as they are available;
- recognised the huge benefits of ERA5 to the meteorological community;
- recognised the value of third-party activities and how they feed into ECMWF output, for example the demonstrated value of ERA5 (e.g. as benchmark) and the potential of prognostic dust aerosol for 2 m temperature forecasts;
- appreciated the value of ECMWF training courses in increasing the benefit of ECMWF forecasts to Member and Co-operating States’ forecasters and end users; welcomed the blended format combining e-learning with classroom face-to-face interactions; but stressed that such face-to-face interactions remain important and should not be reduced further.