The European Flood Awareness System (EFAS) has been operational as part of the Copernicus Emergency Management Service (CEMS) since 2012. ECMWF’s role in EFAS includes running the hydrometeorological computations, archiving the data and disseminating the forecasts. While EFAS development over the years has focused mainly on new products, an upgrade to EFAS v3.3 implemented on 8 October 2019 aimed to improve the operational workflow and the presentation of existing products.
ECMWF has worked with the European Commission’s Joint Research Centre (JRC), which runs CEMS, and the EFAS consortium to redesign the workflow to prepare for future resolution increases and new products. This was an opportunity to improve performance and upgrade the medium‐range forecast products, following feedback from EFAS partners.
Medium-range forecast product upgrade
On the design side, a new forecast layer now provides an overview of ‘reporting points’. These are dynamic points, generated each time a new forecast is produced, in locations where an elevated flood risk is expected in the next 10 days. The reporting points are shown together with locations for which no flood risk is predicted but where hydrological information is shared by EFAS partners (the ‘static points’). This new design gives users a complete and coherent overview, in a single layer, of all locations at which EFAS medium‐range forecasts are accessible. It also enables partners who have shared their hydrological data with the EFAS consortium to monitor the hydrological evolution of their catchments of interest. This was not possible before if the predicted flood signal did not reach an EFAS‐defined threshold. In addition, the way the reporting points are displayed has also been improved. As shown in the figure, stations with an elevated flood risk are quickly identifiable (red and yellow squares), facilitating the work of the forecasters on duty at the EFAS dissemination centres (the hydrometeorological services of Sweden, the Netherlands and Slovakia).
Performance gains
The expected growth in the number of EFAS partners, and thus of reporting points, together with the envisaged increase in resolution of EFAS products motivated a full review of the existing product generation chain to enable greater scalability and prepare for future upgrades. This was also an opportunity to reduce the complexity of the legacy tools that make up the operational suite, which used a mix of programming languages, such as Python, R, PCRaster, C++ and bash scripts. The new product generation is based on a newly developed Python framework called ‘danu’. This framework aims to:
- gather common functionalities of flood forecasting in a shared framework
- improve modularity of the system, allowing easier implementation of new features
- improve maintainability using modern software engineering processes
- promote collaborative development practices between hydrologists and computer scientists
- introduce parallel computing and data hypercubes into the process, preparing the system for the next resolution upgrade.
Using parallelism in compute‐intensive parts of the workflow has led to dramatic performance gains, from 5 times faster for alert generation to up to 50 times faster for ensemble statistics. Porting the multiple processes into a single framework also made it possible to optimise filesystem I/O by avoiding data transfers, which was a major performance bottleneck. These performance improvements were achieved thanks to close collaboration between the scientists that developed the system and computer scientists that reworked it, creating positive feedback where technical developments empower the science to go further. The tools developed in the Python framework ‘danu’ and this successful collaboration between teams are now being applied to other environmental forecast modelling chains, such as those developed for the EU‐funded SMUFF project ‘Seamless probabilistic multi‐source forecasting of heavy rainfall hazards for European flood awareness’.