Driving innovation in data provision – the Data Stores Service

19 May 2026

Abstract digital corridor filled with blue, glass-like structures displaying streams of binary code (0s and 1s).

ECMWF has long-standing expertise in developing advanced software for weather and climate, and extensive operational experience in managing high-demand, time-critical data services.

The ECMWF Data Stores Service (DSS) further enhances ECMWF’s broad portfolio of data services and provides a single, unified gateway to a vast array of weather, climate and environmental data. It supports an ever-widening spectrum of users on their journey from data discovery and exploration to retrieval, analysis, and application.

Built on ECMWF’s expertise, the DSS represents a new generation of data infrastructure designed to facilitate the uptake of data and services by a broad and diverse community of users across domains and levels of expertise. It is continuously evolving through the adoption of cutting-edge technologies – driven by user needs and advances in the ECMWF Software Engine.

The DSS not only delivers robust operational services but also serves as a dynamic platform for innovation, where new data solutions can be rapidly developed, tested and deployed in a cloud-based environment.

Developed from the cloud-based Climate and Atmosphere Data Stores infrastructure designed by ECMWF as part of its role in the EU Copernicus Programme, the DSS now forms a common operational backbone supporting a range of ECMWF data interfaces.

“The DSS provides a single, unified gateway to a vast array of weather, climate and environmental data, supporting users from discovery to application. It is a dynamic platform for innovation, where new data solutions can be rapidly developed, tested and deployed in a cloud-based environment," said Umberto Modigliani, Acting Director of Forecasts at ECMWF.

Supporting users from data discovery to application

The DSS provides a single, harmonised entry point to a distributed set of weather, climate, atmospheric composition, and environmental data sources, enabling seamless access through both user-interactive interfaces and programmatic APIs.

It supports the full user journey – from data discovery and exploration to subsetting, retrieval and downstream use – serving a broad and continuously expanding user community with diverse levels of expertise and technical requirements.

The service operates 24/7, supporting:

over 7,500 active users each day,
around half a million data requests per day,
the delivery of around 100 TB of data each day on average, and
more than 280,000 registered users.

Providing a shared operational backbone

The DSS is implemented as a layered architecture comprising data services, a common software platform and user access interfaces, hosted on ECMWF’s on-premises Common Cloud Infrastructure (CCI).

Schematic of the ECMWF Data Stores Service (DSS) architecture showing data layers, backend services, APIs, and user-facing interfaces.

Together, these layers form a shared, reusable operational backbone that underpins ECMWF’s Data Store interfaces, including the Climate Data Store (CDS) for the Copernicus Climate Change Service (C3S), the Atmosphere Data Store (ADS) for the Copernicus Atmosphere Monitoring Service (CAMS), the Early Warning Data Store (EWDS) for the Copernicus Emergency Management Service (CEMS), and the recently launched ECMWF Data Store (ECDS). While each interface is tailored to the needs and objectives of its respective user community, they are all powered by the same harmonised, centrally managed DSS platform.

The DSS architecture supports access to data hosted on the CCI, data served from the ECMWF exabyte MARS archive, and data brokered from external repositories. Integrating the DSS software and platform services with the ECMWF MARS data archive is strategically critical, as MARS is the primary source of data delivered by the Data Stores and accounts for most user traffic. This physical proximity enables highly efficient, low-latency access to very large data volumes, minimises data movement and duplication, and underpins the high throughput and reliability required for sustained operational delivery at scale.

Designed for growth

Hosting the DSS on ECMWF cloud infrastructure enables the organic and synergistic growth of all its layers. Computing, storage, software services and access components can evolve in concert as user demand, data volumes and functional requirements increase.

This cloud-based deployment model supports the reuse of components across services and enables new capabilities – such as additional access mechanisms, processing options or data services – to be introduced incrementally and consistently across all Data Stores without disrupting existing users.

A core design principle of the DSS is harmonisation and automation. Metadata models, catalogue content, processing options and service configuration are centrally managed and automatically deployed across interfaces. This high degree of automation is essential for operating a continuously evolving portfolio of approximately 175 catalogued data records, most of which support post-processing during retrieval (e.g. format conversion, aggregation, and spatial or temporal subsetting), while maintaining consistency, scalability, and operational robustness.

These harmonised foundations are driven by compliance with the FAIR principles, achieving a 92% score across the Findable, Accessible, Interoperable and Reusable dimensions.

Advanced capabilities

A more recent capability introduced as part of the DSS evolution is the implementation of Analysis-Ready, Cloud-Optimised (ARCO) data services, which provide faster, more efficient data access and analysis for large, multi-dimensional datasets in cloud environments. More than 70 DSS datasets are regularly updated, and ARCO versions in cloud-native formats (such as Zarr) now form an integrated part of the operational data processing pipelines.

The ARCO format provides an additional capability for discrete data access patterns – such as time- or space-chunked data – well suited to analytics, time-series extraction, regional studies, interactive visualisation and machine learning applications that require efficient access to large volumes of structured data.

These capabilities are being consolidated into an operational component of the DSS and consistently reused across Data Store interfaces and federated platforms.

Complementing the managed data services, ECMWF also develops and maintains earthkit, an open-source Python framework that provides reusable building blocks for data access, formatting, processing, analysis and visualisation.

Earthkit is closely aligned with DSS interfaces and ARCO data services, enabling users to move seamlessly from data discovery and retrieval to advanced analysis, application development and interactive visual content.

Together, DSS, ARCO and earthkit lower the technical barrier to working with Copernicus and other Earth system data, and promote reproducible, best-practice workflows across the user community.

A typical data pipeline

Let’s look at a typical data pipeline using the ERA5 reanalysis dataset as an example.

The DSS on-disk MARS data repository includes a subset of the most requested variables from the core MARS archive, which are uploaded daily by automatic suites.

From that subset, 21 variables from ERA5 single levels are currently available via the DSS ARCO data lake, where they are stored as data cubes chunked both spatially and temporally. Storing data in this way enables efficient, on-the-fly, discrete access to specific subsets of data (e.g. time series over a selected location), avoiding the need to scroll through huge datasets. Users can access ARCO data through a user interface and programmatically via APIs and the token access service.

The ARCO data lake repository application provides tiled map services and interactive visual content.

Together with earthkit, these capabilities support advanced user applications, such as ERA Explorer and Weather Replay. Here you can see the power of ARCO data, which enables the ERA Explorer to process and display, for example, grid-point daily, monthly, and annual statistics derived from hourly data, in a matter of seconds.

Screenshot of the Weather Replay app showing global view of Earth centred on Africa and Europe, overlaid with blue weather patterns in a data visualisation interface.

Screenshot of the Weather Replay application. Users are able to visualise hourly weather data since 1940 from the ERA5 reanalysis.

A key element of a wider data ecosystem

The DSS architecture is designed to operate as part of a wider ecosystem of federated platforms.

“As a strategic, cross-cutting infrastructure that drives innovation, the DSS leverages its layered architecture and advanced ARCO data services to transform vast environmental archives into a highly automated, cloud-optimised ecosystem that provides efficient, on-the-fly access to Earth system data at continental scale,” said Angel López Alós, Data Stores Service Manager at ECMWF.

In particular, ECMWF contributes to the Copernicus WEkEO Data and Information Access Service – one of five cloud-based platforms providing access to Copernicus data. ECMWF’s role focuses on the evolution of the Data Access Services layer and the operational management of the interfaces between DSS and WEkEO to harvest catalogue content, retrieve data, and operate and visualise products. These integrations build directly on DSS principles of harmonisation, reuse and automated configuration.

Finally, the Data Stores software infrastructure is fully integrated into the ECMWF Software Engine (the software layer that powers ECMWF services), sharing and reusing components, frameworks and architectural practices. This integration strengthens robustness and maintainability, reduces duplication of effort, and ensures that innovations introduced at the platform level – such as new access mechanisms, ARCO services or automation improvements – can be consistently implemented across all Data Store interfaces and reused more broadly across ECMWF systems and federated platforms.

Together, these elements position the ECMWF Data Stores Service as a strategic, cross-cutting infrastructure that integrates harmonisation, reuse, automation, cloud-based scalability, and proximity to core data assets, enabling reliable, efficient and innovative access to Earth system data at scale.

The ECMWF DSS is a revolutionary, strategic, cross-cutting infrastructure that is transforming how the world interacts with Earth system data. By serving as a single, unified gateway to a vast array of weather, climate and environmental information, it has successfully bridged the gap between complex data discovery and real-world application.