Ongoing research project | 2015 - 2018
EarthServer - Agile Analytics on Big Data Cubes
About the project
The Horizon 2020 funded EarthServer project aims to establish scalable web-based analysis and processing services for multi-dimensional geo-referenced Earth Science data. Big Earth Science data often need extensive processing in order to retrieve meaningful and communicable information for users and decision-makers. The challenge is to extract the right information from petabytes (PB) of raw data. Ideally, data processing and analysis should be performed on server-side and only kilobytes of refined information are downloaded.
By exploring rasdaman, an intelligent Array Database Technology, in combination with standard data access protocols defined by the Open Geospatial Consortium (OGC), EarthServer explores the possibilities and challenges of providing access to data sizes beyond one petabyte of 3-D and 4-D Earth Science datacubes. In the course of EarthServer, large data centres, such as ECMWF, PML, MEEO/ESA, NCI and Jacobs University, aim at setting up web-based services related to water, air, climate and planetary science data.
About ECMWF's role in EarthServer
ECMWF's participation in EarthServer is in the scope of its Scalability programme. MARS is ECMWF's Meteorological Archive and Retrieval System, the world's largest archive of meteorological data. In November 2015, the MARS archive held ~87 PB of data and grew by additional ~3 PB every month. In order for users to fully benefit from the potential of a data volume beyond the PB, it is in the interest of ECMWF as a data provider to minimise the necessary data transport and yet, to provide access to the full range of data and information.
ECMWF will set up a test web service that facilitates climate data access, exploration, analysis and visualisation. The aim is to access more than 1PB of global reanalysis retrieved from ECMWF's MARS archive and to serve it with the help of rasdaman and OGC-based standard data access protocols in order to e.g. visualise the data with NASA WebWorldWind.
In this way, ECMWF data shall become easier accessible to researchers and decision-makers who use GIS standards. We aim to
- minimise data transport in general,
- serve data and information in standard GIS formats, and
- provide web-based data services on-demand.
EarthServer-2 is receiving funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 654367.
EarthServer Project Partners