In November and December 2024, under ECMWF’s Member and Co‑operating State short-term secondment programme (https://www.ecmwf.int/en/about/jobs/application-member-state-short-term-secondment-ecmwf), ECMWF hosted Angel Marčev, a visitor from the Institute of Hydrometeorology and Seismology of Montenegro (IHMS), to run limited-area, high-resolution re-forecasts for the past year. This was to be done using a new configuration of the IHMS operational numerical weather prediction (NWP) system on ECMWF’s high-performance computing facility (HPCF). Daily re-forecasts over a full year, including all seasons, are crucial for evaluating model performance and developing post-processing techniques preceding operational implementation of a new system. During the visit, with support from ECMWF User Services, the model was installed on ECMWF’s HPCF, re‑forecasts for the year December 2023 to November 2024 were completed, and preliminary verification scores were obtained. Montenegro is an ECMWF Co‑operating State, which has started the application process to become a Member State (MS). The visit was an opportunity to explore the benefits of having full HPCF access and preparation for becoming a Member State.
Background and experiment description
IHMS has a long tradition of running limited-area NWP models to provide operational forecasts. Due to Montenegro's very complex orography, the horizontal resolution of current global models, e.g., 9 km for ECMWF’s Integrated Forecasting System (IFS), is insufficient to resolve small-scale weather patterns, especially those related to strong convection over mountainous terrain. Therefore, IHMS significantly relies on high-resolution non-hydrostatic regional NWP to supplement the global IFS in providing reliable local forecasts and issuing weather warnings. The operational NWP system at IHMS is based on the Non-hydrostatic Mesoscale Model (NMM) on E-grid. The dynamical core was developed by the US National Oceanic and Atmospheric Administration/National Centers for Environmental Prediction inside the Weather Research and Forecast (WRF) system known as WRF-NMM. The model scientific documentation by Janjic et al. is available at: https://dtcenter.org/sites/default/files/community-code/hwrf/docs/scientific_documents/WRF-NMM_2018.pdf. One of the main limitations for NWP developments at IHMS is a lack of high-performance computational resources. As an ECMWF Co‑operating State, Montenegro has access to all products from ECMWF’s global operational forecast and to some computing facilities. This includes ECGATE Class Services (ECS) and the European Weather Cloud (EWC), but not ECMWF’s full HPCF, which is made available only to Member States. Current on-premises infrastructure in Montenegro is sufficient for running high-resolution regional models, for example at 1 km over small domains, but it does not enable extensive testing of sensitivity to model configurations and evaluating performance before a specific configuration is deployed to operations. Furthermore, the current infrastructure does not enable the production of long-term re‑forecasts for calculating the model climatology and archiving NWP results for a long time. For this reason, IHMS approached ECMWF asking for support in doing re-forecasts with their new NWP configuration based on WRF-NMM. The re-forecasts of 2024 on ECMWF’s Atos HPCF during the visit covered three regional domains, two of which are shown in the first figure:
- Domain 1: with 3 km horizontal resolution, driven by initial and boundary conditions (IBC) from the IFS
- Domain 2: with 1 km horizontal resolution, nested in Domain 1
- Domain 3: with 400 m horizontal resolution, driven by Domain 2
Running all three WRF‑NMM domains requires 10 nodes on the Atos HPCF to meet the operational requirement of completing one day of forecasts in seven minutes and completing one year of daily forecasts 72 hours ahead in less than one month.
All three domains were run for the full year from December 2023 to November 2024, daily, starting at 00 UTC to 72 hours ahead. Initial and boundary conditions from the IFS were extracted from ECMWF’s Meteorological Archival and Retrieval System (MARS). Hourly model results from all three domains were stored on the HPCF. A subset of fields required for further processing was archived in a Simple Storage Service (S3) bucket of the EWC, using the storage allocation of Montenegro granted to Member and Co‑operating States. These results have been used to evaluate forecast quality and calculate model biases with bias correction for key observation points in Montenegro. The same archive will be used later to apply existing and develop new post-processing techniques based on machine learning to improve high-resolution forecast products further.
Initial verification
Some verification scores have already been produced for all World Meteorological Organization (WMO) stations in Montenegro and observational stations covered by Domain 3 in the surrounding countries. In the second figure, wind speed monthly bias for the airport in Dubrovnik is shown for each WRF‑NMM domain as well as for IFS direct model output. Despite the overall good match with observations, the global IFS tends to underestimate the wind speed due to its coarser resolution and a simplified representation of the complex orography around the observation station, compared to the WRF-NMM models. Negative bias for the high-resolution regional model is clearly reduced, while the Domain 3 configuration, with the highest horizontal resolution of 400 m, provides the best results overall.
As an added value of this visit, Angel improved his knowledge about ECMWF’s HPCF, the EWC, and other computing facilities. He was also able to familiarise himself with software packages such as ECMWF’s workflow manager ecFlow, which can be implemented locally at IHMS to support operational and research workflows.
This work was supported by the German Early Career Fellowship programme and the Senior Research Visitors programme, which funded the secondment, and by the Republic Hydrometeorological Service of Serbia (RHMS), which generously provided the required HPCF resources from its ECMWF Member State allocation.