A nonhydrostatic finite-volume option for the IFS

 

Over the next decade, many aspects of ECMWF’s Integrated Forecasting System (IFS) may need to change in light of the efficiency of higher-resolution global forecasts called for by ECMWF’s long-term strategy. The dynamical core lies at the heart of the model infrastructure. It numerically solves the fundamental governing equations, which take the form of physical conservation laws describing the resolved atmospheric dynamics. The dynamical core is coupled to parametrizations of subgrid-scale physical atmospheric processes and to models of other Earth system components. The current IFS dynamical core depends on the spectral-transform method to solve the governing equations. ECMWF is continuing to develop this dynamical core, which also includes a nonhydrostatic option, to make it as computationally efficient as possible. For added flexibility, it is also developing a new, nonhydrostatic dynamical core which uses the finite-volume method. This ‘Finite-Volume Module’ (FVM) has been shown to perform well compared to the current dynamical core in benchmark tests, and it holds the promise of greater computational efficiency for global nonhydrostatic forecasts at very high resolution run on future exascale high-performance computing (HPC) facilities.

The current dynamical core

The dynamical core of the operational IFS solves the hydrostatic primitive equations (HPEs) using the spectral-transform (ST) technique. It will be referred to as the IFS-ST in this article. The ST technique was introduced in the ECMWF forecasting model in the 1980s. It is still successfully employed today in combination with a semi-implicit semi-Lagrangian (SISL) integration scheme (see Wedi et al., 2015, for details and a comprehensive list of references). Recent advances, such as the cubic truncation and the Fast Legendre Transform, will ensure the computational efficiency of the ST method in the next decade. Nevertheless, there are uncertainties regarding the long-term viability of that method. These uncertainties are rooted in the scalability of the high-volume non-local data communications associated with it. At some point, this could lead to an excessive amount of time spent on parallel communication, preventing the timely delivery of the forecast (Wedi et al., 2015). In response to these issues, ECMWF is widening its options with respect to the dynamical core.

On the one hand, ECMWF is improving the computational efficiency of the ST method and of the nonhydrostatic extension of the IFS-ST that has been kindly made available by Météo-France and the ALADIN consortium. On the other hand, ECMWF is pursuing the development of methods with fundamentally different parallel communication patterns and a complementary nonhydrostatic dynamical core design, such as in the FVM.

The Finite-Volume Module

The Finite-Volume Module of the IFS (henceforth IFS-FVM) is currently under development at ECMWF (Smolarkiewicz et al., 2016, Kühnlein et al., 2018). An important property of the finite-volume (FV) method applied in the IFS-FVM is that solutions to the governing equations are calculated at discrete places on a meshed geometry (Box A). This means that there is a distributed-memory communication footprint that is predominantly local and performed via thin ‘halo’ cells shared with the nearest neighbours. This contrasts with the high-volume non-local communications required in the IFS-ST. In addition to the different communication patterns, the IFS-FVM provides numerous capabilities that are currently unavailable in the IFS. Among the advantages of the FV method are inherently local conservation and the ability to operate in complex mesh geometries (Kühnlein & Smolarkiewicz, 2017). The lack of conservation is a common issue with standard semi-Lagrangian schemes and a shortcoming in the operational IFS, which is suspected to contribute to moisture and temperature biases and to affect the forecast quality, particularly at sub-seasonal to seasonal forecast ranges. The ability of FV methods to operate in complex mesh geometries makes it possible to implement efficient quasi-uniform-resolution meshes that circumvent the polar anisotropy of the classical regular longitude-latitude grids. Furthermore, it enables variable and/or adaptive resolution with locally finer mesh spacings in sensitive regions (e.g. storm-tracks) and hence a more accurate and efficient representation of multi-scale interactions.

​A novel atmospheric dynamical core formulation for NWP represents a long-term development. In addition to the dynamical core itself, it involves various other aspects of the model infrastructure. The design of the IFS-FVM facilitates its incorporation into the existing IFS. Table 1, adapted from Kühnlein et al. (2018), lists the main features of the IFS-FVM on the one hand and the hydrostatic and nonhydrostatic IFS-ST dynamical cores on the other. Despite fundamental differences in the governing equations and the discretisation, the different dynamical cores can share certain features. The most prominent examples are the octahedral reduced Gaussian grid (hereafter referred to as the ‘octahedral grid’) of the operational IFS and the co-located arrangement of all prognostic variables at the same nodes of the grid. This means the FV mesh is developed about the nodes of the same octahedral grid that supports the IFS-ST (Figure 1). The FV mesh is defined in terms of edges connecting the nodes and dual volumes around the nodes (Box A). The use of the same octahedral grid for the different dynamical cores has numerous benefits for the general model infrastructure, data assimilation and model initialisation, as well as comparison studies. In the IFS-FVM, mesh generation, mesh data structures and nearest-neighbour distributed-memory communication are provided by ECMWF’s Atlas framework (Deconinck et al., 2017), which will also provide support for heterogeneous HPC architectures.

To make the IFS-FVM a useful option for global medium-range weather forecasting, it needs to be shown that it can provide forecast quality that is at least comparable to that of the current IFS. Another important question concerns the computational efficiency of the IFS-FVM, which must be sufficient to meet the tight constraints of operational scheduling. In the following, we study the questions of accuracy and computational efficiency, considering test cases of intermediate complexity at the current stage of development. More details about the IFS-FVM model formulation used and the comparison to the IFS-ST can be found in Kühnlein et al. (2018).

Comparison to spectral-transform IFS

Basic evaluation of the IFS-FVM dynamical core has been performed using the baroclinic instability benchmark (see Kühnlein et al., 2018, for a description of the setup). This is a commonly used test to evaluate the performance of NWP models in the large-scale hydrostatic regime. Here, the proven hydrostatic IFS-ST dynamical core that is used for operational forecasting at ECMWF will serve as the reference.

Figures 2 and 3 show solutions for the two dynamical cores using the same octahedral grid and the difference between them. Results are shown for an example forecast at day 10, when a large-amplitude baroclinic wave has developed and has formed sharp fronts in the lower troposphere. The difference plots in Figures 2e,f and 3e,f show that there is very close agreement between the IFS-FVM and the IFS-ST. The solutions show essentially similar phase propagation and amplitude of the baroclinic wave throughout the whole depth of the simulation domain. There are only small differences at a grid spacing of about 62 km (O160 for the IFS-FVM and TCo159 for the IFS-ST) and they become even smaller when reducing the grid spacing to about 32 km (O320 and TCo319, respectively). Overall, the results attest to the high quality of the IFS-FVM dynamical core.

Computational efficiency

Computational efficiency of NWP models is crucial. For current HPC architectures at operational resolutions, the IFS-ST at ECMWF represents one of the most efficient dynamical core formulations for global NWP. The IFS-FVM is envisaged for future applications in the nonhydrostatic regime running on future HPC architectures, but its computational performance on the current HPC facility at ECMWF sheds light on its potential. We have assessed the computational efficiency of the dynamical cores using the dry baroclinic instability benchmark presented in Figures 2 and 3.

The IFS-FVM is a recent development, and considerable progress has been made in terms of its computational efficiency. Over the last two years, the IFS-FVM has been speeded up by at least a factor of seven. This has been achieved by optimising various aspects of the time stepping and by extensive recoding of the algorithms. For details, see Kühnlein & Smolarkiewicz (2017) and Kühnlein et al. (2018).

Figure 4 shows runtimes of the nonhydrostatic IFS-FVM against the hydrostatic IFS-ST, both configured similarly to ECMWF’s current high-resolution forecast (HRES).  The time-to-solution that can currently be achieved with the IFS-FVM is only about twice as long as for the operational hydrostatic IFS-ST. Indeed, the IFS-FVM already compares favourably with other nonhydrostatic dynamical cores. Moreover, the local FV method offers the prospect of better scalability and efficiency with respect to future HPC. Another important aspect is that the IFS-FVM employs smaller time steps (typically smaller by a factor of 5 to 6 compared to the IFS-ST), which can be beneficial for accuracy. Overall, the results highlight the potential of the IFS-FVM to become competitive for operational global weather forecasting.

Moist dynamics and coupling to parametrizations

The results presented above were obtained using the basic dry dynamical cores. At the current stage of development, the IFS-FVM has been extended to moist/precipitating dynamics and coupling to selected IFS physical parametrizations (Kühnlein et al., 2018). Work on coupling to the full IFS physical parametrization package is ongoing. Figures 5 and 6 compare the IFS-FVM to the reference IFS-ST using the baroclinic instability benchmark with moist/precipitating processes. For this particular test, coupling to the operational prognostic single-moment bulk microphysics parametrization of the IFS for large-scale condensation and precipitation is applied, implemented by means of a generic interface suitable for all parametrizations. The current generic interface calls and evaluates the physical parametrizations sequentially at each of the IFS-FVM semi-implicit time steps, which is the standard procedure in the operational IFS-ST. Because of the significant cost of the IFS physical parametrizations (about 30% of the forecast model), the physics–dynamics coupling with the IFS-FVM needs to be revised in order to maintain computational efficiency, given the smaller time steps of the IFS-FVM.

Figure 5 shows an example surface precipitation field at forecast day 10. Five rain bands which are virtually identical for the two dynamical cores appear at the two chosen horizontal grid spacings. The rain bands, which are the result of air being lifted along sharp frontal zones, are essentially identical in phase. This is highlighted in Figure 5 by overlaying the IFS-FVM precipitation rate shading with IFS-ST 0.5 mm/day contours and vice versa. In addition, precipitation rates are overall similar. Figure 6 shows a time series of the minimum near-surface pressure and the area-integrated precipitation rate over the course of a 15‑day simulation. It indicates close agreement in terms of the magnitude and temporal evolution of these quantities, and no systematic biases. Again, the presented results attest to the high quality of the IFS-FVM dynamical core with moist/precipitating dynamics and the coupling interface to physical parametrizations.

The ability of next-generation dynamical cores to produce global forecasts in the nonhydrostatic regime is essential. The IFS-FVM participates in the current round of the Dynamical Core Model Intercomparison Project (DCMIP), in which next-generation nonhydrostatic dynamical cores from various weather forecasting centres and research institutions are compared using common test cases. Figure 7 provides an illustration from a ‘splitting supercell’ test case, run in a small planet configuration at about 1 km horizontal grid spacing. The grid spacing approaches the horizontal extent of convective updrafts so that the experiment is a test case for resolved nonhydrostatic dynamics. The dynamical core solutions feature different responses in small scales depending on the numerical discretisation and filtering mechanism. The properties of different numerical formulations for global atmospheric models in the regime where convection is (partially) resolved will be studied more systematically over the next few years.

Outlook

Supporting substantially higher resolution in global NWP may ultimately demand local numerical discretisations to solve the governing nonhydrostatic equations in a computationally efficient manner. The IFS-FVM successfully implements such a local discretisation and thus complements the operational hydrostatic IFS-ST at ECMWF. At the same time, the IFS-FVM introduces several useful new features into the IFS, such as conservative and monotone advective transport, deep-atmosphere nonhydrostatic governing equations, and fully flexible unstructured FV meshes with optional variable resolution or meshes defined about the nodes of the operational octahedral grid. Furthermore, the recently extended perturbation form of the IFS-FVM equations offers significant potential for long-range predictions (Smolarkiewicz et al., 2019).

As shown in this article, the IFS-FVM has advanced to a stage where it can provide forecast quality comparable to that of the operational ST dynamical core for benchmark tests of intermediate complexity. Substantial progress has been made in terms of computational efficiency of the IFS-FVM dynamical core, and further improvements are in preparation.

Ongoing and future work on the IFS-FVM involves all the technical and scientific aspects of coupling it to the full IFS physical parametrization package and configuring the overall model to run global medium-range re-forecast experiments for comparison with the operational IFS-ST. Further research and development will extend the IFS-FVM numerical schemes; explore multigrid techniques; explore alternative physics–dynamics coupling for the smaller time steps; introduce variable precision; and adopt I/O procedures of the IFS. Tangent-linear and adjoint versions of the nonlinear IFS-FVM model required in the context of 4D-Var data assimilation will be developed.

Adaptation of the IFS-FVM to novel low-energy heterogeneous HPC architectures will be facilitated by the data structures of Atlas, which makes use of specific programming paradigms. This is considered to be a basis from which the IFS-FVM can reach out to extreme scalability and into the realm of global nonhydrostatic convection-resolving weather forecasting.

Figures 1, 2, 3, 5 and 6 reproduced from Kühnlein et al. (2018) under the CC BY 4.0 licence.

---

Further reading

Deconinck, W., P. Bauer, M. Diamantakis, M. Hamrud, C. Kühnlein, P. Maciel, G. Mengaldo, T. Quintino, B. Raoult, P.K. Smolarkiewicz & N.P. Wedi, 2017: Atlas: A library for numerical weather prediction and climate modelling. Comput. Phys. Commun., 220, 188–204.

Kühnlein, C. & P.K. Smolarkiewicz, 2017: An unstructured-mesh finite-volume MPDATA for compressible atmospheric dynamics. J. Comput. Phys., 334, 16–30.

Kühnlein, C., W. Deconinck, R. Klein, S. Malardel, Z.P. Piotrowski, P.K. Smolarkiewicz, J. Szmelter & N.P. Wedi, 2018: FVM 1.0: A nonhydrostatic finite-volume dynamical core formulation for IFS. Geosci. Model. Dev. Discuss., doi:10.5194/gmd-2018-237, accepted.

Smolarkiewicz, P.K., W. Deconinck, M. Hamrud, C. Kühnlein, G. Mozdzinski, J. Szmelter & N.P. Wedi, 2016: A finite-volume module for simulating all-scale global atmospheric flows. J. Comput. Phys., 314, 287–304.

Smolarkiewicz, P.K., C. Kühnlein & N.P. Wedi, 2019: Semi-implicit integrations of perturbation equations for all-scale atmospheric dynamics. J. Comput. Phys., 376, 145–159.

Wedi, N.P., P. Bauer, W. Deconinck, M. Diamantakis, M. Hamrud, C. Kühnlein, S. Malardel, K. Mogensen, G. Mozdzinski & P.K. Smolarkiewicz, 2015: The modelling infrastructure of the Integrated Forecasting System: Recent advances and future challenges. ECMWF Technical Memorandum, No. 760.