Newsletter No. 148 banner

Web standards for easy access to big data

Julia Wagemann
Stephan Siemen

 

Data-processing chain
Data-processing chain. Example of how a WCS can be integrated into the standard web-based data processing chain.

Since May 2015, ECMWF has been part of the EU-funded Horizon 2020 EarthServer-2 project. Our goal as a project partner is to explore standardised web services on top of ECMWF’s MARS archive for efficient access to large data volumes. The project explores two standard data-access protocols defined by the Open Geospatial Consortium (OGC), designed to offer web-based access to multi-dimensional geospatial datasets: (i) Web Coverage Service (WCS) and (ii) Web Coverage Processing Service (WCPS).

What is OGC WCS?

The Web Coverage Service (WCS) is described by the Open Geospatial Consortium (OGC) (2012) as follows:

“The OGC WCS supports electronic retrieval of geospatial data as ‘coverages’ – that is, digital geospatial information representing space/time-varying phenomena”.

Therefore, WCS is a standard data-access protocol that defines and enables the web-based retrieval of multi-dimensional geospatial datasets.

Unlike Web Mapping Service (WMS), another widely-used OGC standard, which returns spatial data as an image, WCS returns data in its raw form, with its original semantics. This allows for further web-based data processing and analysis or the building of web applications.

Useful web links

EarthServer-2 project: www.earthserver.eu.

ECMWF’s role within EarthServer-2: www.ecmwf.int/en/research/projects/earthserver-2 and http://earthserver.ecmwf.int.

WCS protocol: http://www.opengeospatial.org/standards/wcs.

The experience of the first project year shows that WC(P)S can be of great value, especially for developers or scientists who build applications and want to have access to large data volumes but do not want to store all the data on their local discs. WCS and WCPS are machine-to-machine interfaces and therefore aimed at technical data users or developers rather than end users. By facilitating the spatial and temporal subsetting of gridded meteorological data, time-consuming data download and processing is shortened, enabling a greater focus on the interpretation of data.

Multi-dimensional gridded data fields (called coverages) can be accessed and processed in an interoperable way. Technical data users, for example, can integrate a WCS request into their processing routine and further process the data. Commercial companies can easily build customised web-applications with data provided via a WCS. This approach is also strongly promoted by the EU’s Copernicus Earth observation programme, which generates climate and environmental data as part of an operational service. Commercial companies can use the data to build value-added climate services for decision-makers or clients. Web-based data access with the help of a WCS can make ECMWF data more accessible to researchers, technical data users and commercial companies, within the MetOcean community and beyond.

Test users welcome

Two outreach events within the past year showed that there is growing interest in ECMWF data being accessible via web services. In January 2016, ECMWF hosted a hackathon that aimed to improve the Global Flood Awareness System (GloFAS). Part of the 3.5 TB of data that had been prepared for the event was served with the Web Coverage Service that is being set up at ECMWF as part of the EarthServer-2 project. Additionally, raw datasets were offered for download via an FTP server. Four out of five teams took advantage of the facilitated data access and integrated the WCS data retrieval into their applications. By exploiting the WCS, the teams were able to devote most of their time to building functional applications rather than wasting it on data download and pre-processing.

In April 2016, the EarthServer-2 project hosted a splinter meeting at the General Assembly of the European Geosciences Union (EGU) in Vienna. For this event, all EarthServer-2 service partners built a custom web-client that visualises global data fields with NASA WebWorldWind, a three-dimensional virtual globe API, and allows the on-demand retrieval of time series information of individual geographical points. The demonstration at the meeting showed that it is simple to build customised web-applications on top of a WCS. Participants in the meeting were keen to use ECMWF’s WCS service to gain access to climate reanalysis data.

We are planning to grant access to the WCS to selected test users. If you are interested in using the WCS service, please send an email to earthserver@lists.ecmwf.int and we will get in touch with you.