ECMWF scientists are exploring how machine learning models trained directly on Earth system observations can reconstruct the historical state of the atmosphere from observations alone.
The work, described in a preprint available on arXiv, opens exciting opportunities for how observation-driven artificial intelligence (AI) prediction could be used to complement future global reanalysis production.
Reanalysis datasets provide a consistent reconstruction of the atmosphere over many decades. Traditional approaches to producing these datasets combine observations from satellites, weather stations, aircraft, radiosondes and other observing systems with numerical weather prediction models through data assimilation.
The resulting gridded datasets have been used across weather and climate science, from monitoring long-term changes in the climate system to analysing individual extreme events.
ERA5, produced by ECMWF within the framework of the EU’s Copernicus Climate Change Service, has become a cornerstone of weather and climate research.
It is widely used by researchers, public authorities and businesses around the world and also provides the training data for most of today’s leading machine learning weather forecasting systems.
As highlighted recently, the rapid development of AI-based forecasting has been made possible to a large extent by the open and public availability of consistent, high-quality reanalysis data such as ERA5.
Against this backdrop, ECMWF scientists are exploring whether machine learning models can be trained directly on observations as a complementary approach to established reanalyses such as ERA5.
The research used AIFS-DOP, a version of ECMWF’s Artificial Intelligence Forecasting System (AIFS) that is trained directly on observations. The model does not use ERA5 or another reanalysis as a training target. Instead, it learns to reconstruct a complete atmospheric state from sparse and unevenly distributed observations.
The work was carried out using Anemoi, the open-source machine learning framework developed by ECMWF together with national meteorological services across Europe.
Anemoi provides the tools needed to build AI-ready datasets and to train and deploy machine learning models for operational weather applications. It already underpins the development of global and regional AI forecasting systems, and this study extends its use to a new area: the generation of global reanalysis datasets.
Reconstructing more than four decades of weather
As a proof-of-concept, a prototype reanalysis has been produced covering the period from 1981 to 2022 and provides estimates of the atmospheric state every six hours on a global grid with a resolution of approximately 112 km.
For each analysis time, the model uses observations from the preceding 30 hours, allowing information from earlier observation windows to contribute to the reconstructed state. The model was trained on a carefully curated dataset of conventional and satellite observations prepared using the Anemoi framework.
Once trained, the prototype generated the complete 42-year dataset in the course of a single working day, illustrating the potential for observation-driven machine learning to offer faster and more flexible approaches to producing complementary reanalysis datasets.
Capturing the large-scale atmospheric patterns
ECMWF scientists evaluated how well the machine-learning reanalysis reproduces the mean structure of the atmosphere and its variability across different timescales.
It was shown that the AIFS-DOP reanalysis captures the large-scale distribution of temperature and wind, including the subtropical jets, the seasonal movement of the circulation and the broad structure of the Hadley cells. Zonal-mean temperature differences from ERA5 are generally small, although larger differences remain in some polar regions.
The seasonal migration of the Intertropical Convergence Zone is also represented, together with features such as the South Pacific Convergence Zone and areas of subtropical divergence. The comparison shows that the model can reconstruct these large-scale circulation patterns from sparse observations, including in regions where direct measurements are limited.
The prototype machine-learning reanalysis also reproduces important forms of atmospheric and oceanic variability. Northern and southern hemisphere storm tracks appear in the expected locations and with broadly realistic intensity. El Niño and La Niña events are represented through sea-surface temperature anomalies and the accompanying changes in the tropical circulation.
Comparison of northern and southern hemisphere storm tracks in the machine-learning reanalysis AIFS-DOP and ERA5 (top row shows December–January–February (DJF), bottom row June–July–August (JJA), aggregated over 2002 to 2021). The machine-learning reanalysis captures the spatial distribution of the storm tracks and the change in their intensity with season.
On longer timescales, the effects of major volcanic eruptions, variations associated with El Niño and La Niña, and the long-term warming of the troposphere and near-surface land areas are being well captured.
Historical events and physical consistency
Several well-known weather events, including the Great Storm of 1987, the Antarctic polar-vortex split of 2002, an atmospheric-river event in 2021, and Hurricane Irma in 2017, were also analysed.
The large-scale structures of these events are reproduced in the machine-learning reanalysis. Some features are weaker or smoother than in ERA5, particularly where the current resolution limits the representation of small-scale processes. Hurricane intensity, for example, is underestimated, as would be expected for a grid spacing of approximately 112 km.
Machine-learning reanalysis (AIFS-DOP, left) compared with ERA5 (right) for four historical weather events: a) the Antarctic polar-vortex split (2002), b) the Great Storm of 1987, c) an atmospheric river event (2021) and d) Hurricane Irma (2017). The large-scale structures of these events are reproduced in the machine-learning reanalysis.
The team also assessed whether the reconstructed fields are physically coherent. Numerical weather prediction models impose physical relationships through their governing equations. In an observation-driven machine learning system, these relationships have to be learned from the observational data.
In particular, the team examined the balance between wind and geopotential fields, including the variation of the Coriolis effect with latitude, and how temperature, wind and geopotential anomalies vary together around large-scale weather systems. The results show that AIFS-DOP reanalysis reproduces many of the expected dynamical relationships and spatial patterns. This provides encouraging evidence that the system has learned a physically meaningful representation of the atmosphere.
Comparison with independent observations
The prototype reanalysis was also evaluated against observations that were not used for the training or assimilated into ERA5.
For upper-level winds, errors are close to those of ERA5 when the datasets are compared at a consistent resolution. For some surface variables, the error characteristics lie between those of ERA-Interim and ERA5, ECMWF’s fourth- and fifth-generation reanalyses.
There are areas where further work is needed. Relative humidity is too high in some very dry parts of the atmosphere, including the polar regions and the stratosphere. Small-scale kinetic energy is reduced compared with ERA5, and some aspects of the meridional circulation are less well represented. These results will help guide improvements to the training data, loss functions, model architecture and resolution.
Opportunities for future reanalysis production
This work represents an important step in exploring how machine learning could be used for producing global reanalyses.
Operational reanalyses such as ERA5 and ERA6 – which is currently being produced by ECMWF on behalf of the Copernicus Climate Change Service (C3S) – will continue to provide the trusted foundation for weather and climate applications, as well as the high-quality training data underpinning today’s leading AI weather forecasting systems.
The results shown here open exciting opportunities to explore how observation-driven machine learning could complement future operational reanalysis production, supporting faster experimentation and more flexible approaches to generating Earth system datasets.
They also demonstrate the breadth of the Anemoi framework. Developed to support operational machine learning across the European meteorological community, Anemoi now underpins applications ranging from global and regional forecasting to the exploration of machine-learning-based reanalysis.