ECMWF maintains a comprehensive range of verification statistics to evaluate the accuracy of its forecasts. Each year, a summary of verification results is presented to ECMWF's Technical Advisory Committee (TAC). Their views about the performance of the operational forecasting system in 2024 are given in the box.
In 2024, upper-air ensemble forecast (ENS) skill reached a new high point due to the positive effect of Cycle 48r1 of the Integrated Forecasting System (IFS), which was implemented on 27 June 2023 and which included, among other changes, an increase in the horizontal resolution of the ENS from 18 to 9 km. In Europe, for example, the 12-month average lead time at which the Continuous Ranked Probability Skill Score for 850 hPa temperature drops below 25% exceeded 10 days for the first time (red curve in the Figure). More generally, ENS scores of upper-air variables improved by 1% to 3% in the extratropics, and surface scores improved by 2% to 6% after Cycle 48r1 was implemented. This has enabled ECMWF to maintain and consolidate its overall lead for upper-air parameters in the medium range.
For 2 m temperature and 10 m wind speed, the number of large errors in the ENS in the extratropics has decreased by 5–10% over the last 12 months. ENS precipitation forecasts have improved by about a quarter of a day in terms of lead time in the medium range. Despite these improvements, ECMWF is currently not leading in terms of surface forecast skill, especially at shorter ranges. However, Cycle 49r1, which was implemented on 12 November 2024 and includes, among other changes, 2 m temperature data assimilation in 4D-Var and the use of the Stochastically Perturbed Parametrizations (SPP) scheme, is expected to further reduce errors in 2 m temperature, 10 m wind speed and, to a smaller degree, precipitation.
The performance of the DestinE Global Continuous Extremes Digital Twin (DT) run at 4.4 km resolution is very similar to that of the ENS control forecast for upper-air variables, and 1–3% better for surface variables, except for precipitation, where the larger variability at small scales makes it more difficult to see gains in realism reflected in the overall point statistics.
Tropical cyclone (TC) forecasts by the IFS have improved as well, such that in 2024 track forecasts in the Atlantic as well as forecasts of TC genesis compared favourably to those of other global models.
ECMWF's experimental in-house machine-learning (ML) forecasts have progressed substantially in 2024, with the deterministic Artificial Intelligence Forecasting System (AIFS) now ranking above, or very close to (depending on the parameter), other ML models both for upper-air and surface forecasts. The first version of an ensemble AIFS, which started to run experimentally in June 2024, shows both forecast skill and spread-error match comparable to the IFS.
The sub-seasonal forecast from ECMWF ranks first overall when compared to other global sub-seasonal forecast models, although the statistical significance of the lead drops when going from week 2 to weeks 3 and 4.
ECMWF's seasonal forecast gave very good predictions of the transition in 2024 from El Niño back to more neutral conditions, both at seasonal and annual timescales. Record-breaking heat in parts of central and southeastern Europe in summer 2024 was well indicated, while the cold conditions in northwestern Europe, especially in early summer, were not so well captured.
Verification of Copernicus Atmosphere Monitoring Service (CAMS) ozone profiles against sondes in the northern extratropics has shown the highest scores so far. The latest World Meteorological Organization (WMO) quarterly air quality model intercomparison for North America (see https://hpfx.collab.science.gc.ca/~svfs000/na-aq-mm-fe/dist/) indicates that CAMS is leading among global air quality models for PM2.5 and is approximately on a par for ozone and NO2.
The complete set of annual verification results is available in ECMWF Technical Memorandum No. 918 on 'Evaluation of ECMWF forecasts', downloadable from https://www.ecmwf.int/en/publications/technical-memoranda.
The following are other sources of information about verification and forecasting system changes.
- Verification as part of ECMWF's charts page: https://charts.ecmwf.int
- WMO inter-comparison of global model forecast skill: https://wmolcdnv.ecmwf.int
- WMO ocean wave model intercomparison results: https://confluence.ecmwf.int/display/WLW/WMO+Lead+Centre+for+Wave+Forecast+Verification+LC-WFV
- List of ‘Known IFS Forecasting Issues’: https://confluence.ecmwf.int/display/FCST/Known+IFS+forecasting+issues
- IFS cycle changes since 1985: http://www.ecmwf.int/en/forecasts/documentation-and-support/changes-ecmwf-model
Assessment of ECMWF’s Technical Advisory Committee, 10–11 October 2024
With regard to its overall view of the performance of ECMWF's operational forecasting system, the Committee:
a) congratulated ECMWF on the positive benefits of 48r1, noting that impacts of 48r1 are most marked in ENS surface parameters and upper air and least marked in HRES surface parameters, validating the decision to increase resolution;
b) noted 49r1 will bring further improvements with impacts on surface parameters in particular, especially 2 m temperature and 10 m wind speed, whilst recognizing some degradation at 50 hPa and in the tropics;
c) congratulated ECMWF on maintaining their lead over other centres in upper air scores, in particular noting a new high point in some scores due to the introduction of 48r1 and a clear signal for an increase in skill relative to ERA5;
d) noted lower summer 2024 scores compared to recent summers is linked to natural variability, with other centres and AI models showing a similar decrease in scores;
e) noted some improvement to ENS scores for surface parameters, for example in fraction of large errors in 2 m temperature and 10 m wind speed, brought about by 48r1;
f) welcomed improvements in scores for surface parameters in the shorter range, noting that the difference here between ECMWF and those centres who are leading in this respect has narrowed;
g) congratulated ECMWF on maintaining its long-term lead over other centres for significant wave height and its lead for peak period scores;
h) noted that although scores for tropical cyclone position were not as good as in 2023, comparison with ERA5 reveals this to be related to natural variability; additionally, the match between error and spread was good at days 3 and 5;
i) noted improved EFI verification scores for 10 m wind speed and 24 hour precipitation and encouraged further exploration of 2 m temperature skill in extreme situations;
j) welcomed the addition of AI models to this assessment;
k) congratulated ECMWF on the progress of AIFS, which has shown very marked improvement since summer 2023, and is now leading other AI models for some parameters whilst out-performing HRES for certain surface parameters with proposed development of additional surface parameters welcomed;
l) welcomed the inclusion of DestinE continuous extremes digital twin model evaluation, noting this model has been running routinely since August 2023;
m) noted DestinE shows very similar upper air skill to HRES and greater skill for surface parameters, recognizing that this should be expected given DestinE has a higher horizontal resolution than HRES;
n) welcomed the inclusion of hydrological model evaluation;
o) congratulated ECMWF on successful implementation of EFAS5 in September 2023, noting that in simulation mode this model performs considerably better than previous versions mainly due to increases in temporal and spatial resolution and an increase in the number of stations used in system calibration;
p) noted improvement in sub-seasonal scores relative to persistence for week 2 and limited clear signal for improvement in weeks 3 and 4;
q) welcomed the new comparison of sub-seasonal forecasts from different forecast centres, including introduction of a fair measure to adjust scores for ensemble size;
r) congratulated ECMWF on their lead in scores for sub-seasonal forecasts across all domains and parameters, noting sharp drop in skill in the extratropics from week 1 to week 2 and lead being less strong in weeks 3 and 4;
s) recognized that the seasonal forecast for the tropical Pacific captured the onset of last year's El Niño and its subsequent evolution to more neutral conditions well, even if the magnitude of the El Niño was overly amplified, noting this is a known bias; noted, also, the signal for El Niño onset was seen in the 13-month seasonal forecast plume, indicating longer-range predictability;
t) acknowledged that the seasonal forecast for winter 2023–24 highlighted warmer-than-average conditions over the northern hemisphere; that the persistent smaller-scale cold anomaly over Scandinavia was not captured is unsurprising given this was hard to predict at even two-week lead times;
u) noted that the seasonal forecast for summer 2024 captured the generally very warm conditions globally, for example showing some signal for particularly hot conditions over southern and eastern Europe but the cooler anomaly extending from the far northwest of Europe to the far west of Africa was not captured nor were some polar anomalies, though the latter is normal;
v) appreciated the continued development of new diagnostics and products and very good support ECMWF provided to Member and Co-operating States over the last year, with engagement via many mechanisms including online support, the annual UEF, online seminars, site visits and meteorological representatives at Member States; the 2025 UEF in Bologna celebrating ECMWF's 50th anniversary is eagerly anticipated.