© sasha85ru / iStock / Getty Images Plus
Authors: Mathilde Leuridan, James Hawkes, Caragh Bradley, Adam Warde, Peter Tsrunchev, Samet Demir, Matthew Griffith, Tiago Quintino, Umberto Modigliani
Weather and climate datasets are growing rapidly in size and complexity.
ECMWF already produces hundreds of terabytes of data each day to deliver global predictions and monitor the Earth system for its Member and Co-operating States.
Through the Digital Twins within the European Commission’s Destination Earth (DestinE) initiative, ECMWF and its partners are pushing the limits of Earth-system simulation, increasing resolution and generating petabyte-scale datasets.
ERA6, the next climate reanalysis produced by ECMWF for the European Union’s Copernicus Climate Change Service (C3S), will create a record of past weather and climate amounting to ~25 pebibyte (PiB).
All of this data has enormous value, but only if people can reach the part relevant to their work.
Polytope is ECMWF’s data-access system for extracting specific features directly from large datasets, allowing users to retrieve just the data they need for a location, region, trajectory or time series directly from the underlying datacubes.
By bringing the requested information to the user without the surrounding data, Polytope can substantially reduce transfer volumes and simplify workflows.
Polytope enables flexible extraction of data from multidimensional datasets.
Reduced data volumes and faster workflows
Weather and climate data is typically organised as high-dimensional datacubes – a way of organising data across multiple dimensions.
Alongside the spatial grid, their dimensions include date and time, forecast step, vertical level, parameter and ensemble member. This structure suits the way model output is produced and stored, but it does not always match the way users work with the data.
ECMWF services already allow users to select parameters, levels and forecast times. Once a field has been selected, however, many questions concern only a small part of it: the values within a region, at selected locations, or along a route. The difficulty lies in the mismatch between the shape of the question and the shape of the data that can be retrieved.
Polytope’s feature extraction capability addresses this mismatch by returning data that more closely fit the analysis. Researchers and operational services can receive the values they asked for without first transferring the surrounding grid points and removing them locally.
For example, an operational service producing ensemble meteograms for 500 locations might request one parameter for 100 forecast steps across ECMWF’s 51 ensemble forecasts. The required values are spread across 5,100 global O1280 fields occupying around 65 gibibyte (GiB). Precisely extracting the 500 locations produces a result which is just 10 MB, vastly reducing the data cost.
Similarly, a climate researcher studying heatwaves might request hourly 2-metre temperature at 500 locations over twenty years. This produces around 88 million values, or roughly 350 MB before metadata and encoding.
Those values are distributed across global fields from the DestinE Climate Change Adaptation Digital Twin (Climate DT) occupying around 770 GiB. Feature extraction turns the request into a collection of point-series rather than twenty years of global maps.
National meteorological services often work with areas that follow political or physical boundaries. They may need forecast data for an administrative region when issuing warnings, or precipitation across a river catchment when assessing flood risk. Rectangular geographical requests are a poor fit for either.
Greece, for example, occupies only about a quarter of its smallest latitude–longitude bounding box. Roughly three-quarters of the rectangle lies outside the country.
Other applications follow something that moves. A forecaster may want conditions along the predicted track of a tropical cyclone. Aviation services need forecasts along flight paths, including changes in altitude, while marine researchers may be interested in conditions along a ship’s route. Retrieving the box surrounding the complete path also captures all the unused locations, times and vertical levels between its outer limits.
Vertical profiles present a related problem. A forecaster may need conditions through the atmosphere above a station, perhaps repeated over several forecast times. The required information forms a narrow column through the data, yet conventional retrieval can involve complete horizontal fields for every requested level.
In each case, feature extraction returns data closer to the form needed for analysis: selected locations, an irregular region, a profile or a trajectory. This reduces network transfer and temporary storage, but the main difference for users is in the workflow. They can begin working with the requested information without first downloading, cropping and discarding a much larger volume of data.
Getting from big data to useful answers
Making this efficient workflow possible requires a connection between the information users ask for and the way those values are held in the archive. Polytope provides that connection, translating a scientific description into precise instructions for locating and extracting the data.
This translation must operate at ECMWF’s scale. New fields arrive continuously from forecasts and Earth-system simulations, while requests can cut across the data in many different directions. Polytope calculates the required route through the datacube when each request arrives.
A Polytope request combines two descriptions. The semantic MARS selection answers which data is required, using scientific properties such as dataset, parameter, level, date, forecast step and ensemble member.
The feature specification answers which values within that data should be extracted. Users do not have to determine which files contain those values, how the grid is indexed or which parts of the files must be read: Polytope performs that translation behind the scenes.
Examples of Polytope requests, showing how users can extract data for a specific location (left) or a geographic area (right) without needing to know how the data are stored in the archive.
This separation between scientific intent and technical implementation follows the wider ECMWF data philosophy. The MARS language provides a stable scientific vocabulary, while services such as Polytope determine how to fulfil the request using the available infrastructure.
“Polytope lets users describe the information they need in the language of their own problem, rather than adapting their work to the structure of the archive,” said Mathilde Leuridan, Research Software Engineer at ECMWF.
Polytope builds a view of the relevant datacube and works through its dimensions one at a time. It repeatedly slices the requested shape against the coordinates available on each axis, translating the result into fields and ranges of grid-point indexes. Where possible, neighbouring indexes and repeated selections are combined to produce a compact extraction plan. The multidimensional shape being processed is the polytope from which the system takes its name.
GribJump applies this plan to ECMWF’s CCSDS-packed GRIB messages. It uses the selected indexes to locate the corresponding compressed blocks and reads the required byte ranges from storage. This also improves on conventional GRIB subsetting, where a complete field generally has to be read into memory and unpacked before unwanted values can be removed.
“GribJump minimises input/output (I/O) by reading only the bytes corresponding to the requested region, rather than the entire GRIB field,” said Caragh Bradley, Research Software Engineer at ECMWF.
Polytope uses a mapping function (f) to translate geographic coordinates (latitude and longitude, lat and lon) into byte locations within a GRIB file, allowing only the required data to be read from storage.
Polytope then combines the extracted values with their coordinates and scientific metadata and returns them as Open Geospatial Consortium (OGC) CoverageJSON. The result is self-describing and ready for further use, while the source data remain in their original compressed form.
This direct access sits alongside other ways of preparing data for particular workloads. Stable, repeated access patterns, such as machine-learning training, can benefit from a chunked representation such as Zarr. The trade-off is that the complete dataset must be read, transformed and stored as another copy, with each additional chunk layout adding another copy.
These approaches can be described as “twisting” a datacube into a prepared orientation or “jumping” through its existing layout. Both have their place: prepared layouts for repeated access patterns, and Polytope for more varied requests against the primary data without any data-copying overhead.
Feature extraction at operational scale
Polytope is already in routine use within DestinE. The service is deployed operationally in LUMI, Leonardo and MareNostrum 5, three EuroHPC sites forming part of the shared European infrastructure on which DestinE relies. Across these deployments, Polytope handles around 1.55 million retrieval jobs and transfers approximately 250 terabytes (TB) in a typical month.
Around 30% of users employ feature extraction. Although it accounts for roughly 70% of retrieval jobs, about 1.1 million requests each month, its results make up only around 24 TB of the monthly transfer.
At the level of an individual request, the difference is easier to see. An average feature-extraction request returns 22 MB. Delivering the same selected values through full-field retrieval would transfer an estimated 4.6 GB, more than 200 times as much data.
On a 100 Mbit/s broadband connection, the network transfer would take around two seconds for the feature result, compared with over six minutes for the corresponding complete fields.
Across more than a million requests, this difference quickly accumulates. The 24 TB of feature-extraction results delivered each month represents around 5 petabytes (PB) when expressed as full-field retrievals.
Adding the 226 TB already transferred through full-field requests gives an estimated monthly total of 5.2 PB if all requests were served that way. Polytope instead transfers around 250 TB, avoiding close to 5 PB of data movement each month – a reduction of about 95%.
“The operational figures show that feature extraction works efficiently at scale. Polytope delivers the information users request with a fraction of the data movement, increasing the practical value of weather and climate services to society,” said James Hawkes, Data Infrastructure Services Team Leader.
The saving applies throughout the workflow. The DestinE Data Lake does not have to read and serve those additional petabytes, and the network does not have to carry them. Users avoid waiting for, storing and processing data that they would subsequently discard.
Reducing this storage, transfer and processing also lowers the associated energy use.
The Digital Twin data remain distributed across the three sites, close to the systems that produced them. Polytope sends users the selected results rather than the much larger fields surrounding them.
Extending Polytope’s reach
The next step is to apply the same approach to the services at the heart of ECMWF’s weather and climate missions. ECMWF is now bringing Polytope feature extraction into the time-critical systems that deliver its operational forecasts. By reducing data transfer, feature extraction can shorten the path from model output to operational decisions, helping users make better use of increasingly detailed predictions.
The same capability is planned for ERA6, extending selective access from real-time forecasts to climate reanalysis. Here, the challenge is the scale and depth of the dataset.
Feature extraction will help researchers work with long time series and regional evidence from a roughly 25 PiB reanalysis without retrieving all the surrounding data. This will make ERA6’s consistent record of past weather and climate easier to use for climate monitoring, research, adaptation and mitigation.
“For the future of ECMWF services, the challenge is not only to produce more advanced data, but to make that data usable, timely and relevant for every user who depends on it,” said Umberto Modigliani, Deputy Director of Forecasts at ECMWF.
The Polytope feature-extraction library, GribJump and the polytope-server components are open source and open to co-development. The modular stack can be adapted to different archives, forecasting systems and grids, and deployed close to data across distributed computing environments. MeteoSwiss is already deploying Polytope for feature extraction from its real-time ICON forecasts.
Meanwhile, standardisation will make it easier to use the same approach across different services. Polytope already returns extracted results as OGC CoverageJSON, while an OGC Environmental Data Retrieval API is in pre-production, built on top of Polytope.
Through the World Meteorological Organization’s Study Group on Future Data Infrastructure (WMO SG-FIT), ECMWF is also helping to shape the future of these standards for the meteorological community.
Polytope changes both the user experience and the economics of working with very large environmental datasets. Feature extraction makes more data useful by moving less of it.
Further reading
For more information on how ECMWF’s data, evolving infrastructure, open data, and AI-ready systems are reshaping access to weather and climate information:
- Why ECMWF needs both supercomputing and cloud services
- More than data: services enabling Europe’s growing user community
- Driving innovation in data provision – the Data Stores Service
- Data without friction – ECMWF’s multi-faceted approach to improving data usability
- Data sovereignty in practice: ECMWF infrastructure for European services
- Minutes matter: how ECMWF delivers time-critical data at global scale
- Data friction and the user experience: a framework for improvement
- The living archive: inside ECMWF's exabyte-scale meteorological data repository
- Powering the AI weather revolution: from ERA5 to AI-ready pipelines
- ECMWF's data infrastructure and services: ready for the era of Common European Data Spaces