Twenty years of the Framework for Member State time-critical applications

In 2004, ECMWF's Council approved the 'Framework for Member State time-critical applications'. The Framework allows ECMWF Member and Co-operating State users to run time-critical work at ECMWF with varying levels of monitoring and support. Following the 20th anniversary of the approval of the Framework, we review why it was created, describe the applications that have been supported under it, and look at how it has evolved over the years.

Before the time-critical Framework

Since the introduction of the dedicated Member State Unix system in the mid-1990s (Dando et al., 2023), Member State applications running on ECMWF's computing systems have needed to access data produced by the latest ECMWF operational analysis or forecast in near real-time. Initially, to satisfy this requirement, a mechanism called 'job submission under SMS control' was developed. It ran on the Member States' general-purpose server (ecgate) and made use of SMS, the Supervisor Monitor Scheduler, to manage the job submission. Users provided a job script and indicated at which stage of the ECMWF operational forecast production they wanted the job to be executed.

This mechanism allowed users to make effective use of ECMWF by accessing data produced by one of the ECMWF models as soon as possible after production. Typically, the data were post-processed on ecgate to create tailored products, which were then transferred to the users' local systems. By 2004, about 300 jobs belonging to approximately 60 Member State users were being run each day under the mechanism.

Although this mechanism worked well, there were limitations. In particular, the service only provided a mechanism to submit the users' jobs. If the job failed for any reason, including temporary system issues such as a file system being full, then it was the users' responsibility to notice this, fix any issues with the script, and resubmit.

For more complex applications initialised by output from ECMWF models, which needed to run routinely and in a timely fashion on ECMWF systems, different ad-hoc technical solutions were used. In some cases, such as the prediction of ocean waves (which became a core ECMWF activity on 1 January 1999) and the Boundary Conditions (BC) for Limited Area Modelling, ECMWF Optional Projects were used. Others, such as COSMO-LEPS (Consortium for Small-Scale Modelling – Limited-Area Ensemble Prediction System), which started to run at ECMWF in November 2002 (see Montani et al., 2003, for details), ran under SMS suites in a similar way to ECMWF's operational suite.

As more requests were received – notably from the Norwegian Meteorological Institute to support a semi-operational run of its TEPS (Targeted Ensemble Prediction System) suite, and from the Hellenic National Meteorological Service to support its limited-area modelling forecasting activities during the Olympic and Paralympic Games in 2004 – it became clear that a more formal framework for managing time-critical activities was needed.

The Framework for Member State time-critical applications

Recognising the need to overcome the limitations of the job submission under the SMS control mechanism, and to offer a service better suited to the more complex applications being requested, a Framework for Member State time-critical applications comprising three options was proposed (see Box A):

Simple time-critical job submission monitored by ECMWF
Member State suites monitored by ECMWF
Member State suites managed by ECMWF.

The Framework was discussed by ECMWF's Technical Advisory Committee in October 2004 and approved by Council in December 2004. It has been in use ever since.

A

The Framework in detail

Option 1: Simple time-critical job submission monitored by ECMWF:

Suitable for single jobs or simple jobs with multiple steps.
Available to all registered users – no formal request needed.

Option 2: Member State suites monitored by ECMWF:

Suitable for more complex applications comprising several interdependent tasks.
Suites monitored by ECMWF with second- and third-level support provided by the Member States.
Requested by the Technical Advisory Committee (TAC) representative of the relevant Member State.

Option 3: Member State suites managed by ECMWF:

Application developed, tested and maintained by the Member State.
ECMWF monitors the suite and provides second-level on-call support.
It must be possible to test the application using ECMWF pre-operational (e-suite) data.
Third-level support is provided by the Member State.
Requested by the TAC representative of the relevant Member State.

Time-critical option 1

Option 1 of the time-critical Framework (TC-1) provides a simple job submission mechanism, which replaced the 'job submission under SMS control'. TC-1 is available to all registered Member and Co-operating State users with access to ECMWF's high-performance computing facility (HPCF) and ECGATE Class Service (ECS). It is suitable for single jobs or jobs comprising multiple steps with straightforward interdependencies.

To support TC-1, an enhancement of the ECaccess batch system was implemented. The ECaccess command line tools could already be used to submit jobs to the batch systems on ecgate and the HPCF, and a retry mechanism was already available for file transfers using ectrans. This retry mechanism was extended to the batch jobs and a new concept of 'events' or 'notifications' was introduced.

The events are linked to the ECMWF operational dissemination schedule and are triggered by tasks in the ECMWF operational suite. Users subscribe their jobs to the events, and the jobs are held in standby mode until the operational suite sends an event notification. When the notification for an event is received, ECaccess submits all the jobs subscribed to that event and creates a new job entity in standby mode, ready for the next notification of the event. Users can submit their jobs to the TC-1 mechanism either from the command line or using the ECaccess web interface (see Figure 1).

**FIGURE 1** A screenshot of part of the ECaccess web interface for batch job submission. The user can enter the script to be run in the box at the top or upload from a file. The events to which the job should be subscribed can be selected from the notification list, part of which can be seen at the bottom of the page.

To enable ECMWF 24x7 shift staff to monitor the TC-1 jobs, the web interface shown in Figure 2 was developed. The interface alerts staff when jobs fail and provides an option to try to re-run them.

**FIGURE 2** Web interface for monitoring time-critical option 1 jobs. The top part of the interface shows a summary of the status of all jobs for each event grouped by specific forecast runs. Here only those associated with the 00 UTC forecast runs are shown. The lower part shows the status of all jobs associated with a specific event. Job output can be viewed and any failed jobs re-run through the interface.

Over the years, the 'Simple time-critical jobs' service has proved to be very popular. Today, there are about 150 different users running 1,800 jobs on average per day. Each job is subscribed to one of 100 defined events. Many use the service to access real-time data, using the Meteorological Archival and Retrieval System (MARS) to create bespoke products and charts; run trajectory computations; make statistical analyses of ensemble forecasts (ENS); provide data for customers of national meteorological services; or even to trigger suites running under option 2 of the TC Framework. One particularly popular use is to download ENS meteograms for user-specified locations.

Time-critical option 2

Option 2 of the time-critical Framework (TC-2) is appropriate for running more advanced applications comprising several tasks with potentially complex interdependencies. Typically, TC-2 is used for running limited-area models (LAMs), such as HARMONIE, COSMO or ICON, with initial and lateral boundary conditions (LBCs) provided by ECMWF's Integrated Forecasting System (IFS). Workflows running as TC-2 are monitored by ECMWF's 24x7 shift staff, who provide first-line support. Jobs submitted have access to high-priority batch queues on the ECMWF HPCF to ensure they execute in a timely manner. Figure 3 shows currently running TC-2 suites viewed in ecFlowUI.

**FIGURE 3** The TC-2 suites active in November 2024, viewed in ecFlowUI.

The first activities to run as TC-2 applications

In July 2005, COSMO-LEPS, which had been running on the ECMWF HPCF since November 2002, became the first suite to be declared officially as a TC-2 application. The application was developed and maintained by the weather service of the Italian Emilia-Romagna region, Arpae-SIMC, with the aim of providing high-resolution mesoscale ensemble forecasts of localised high-impact weather events. Initially, it ran a control forecast plus a ten-member ensemble based on the 'Lokal Modell' with initial conditions and LBCs from ECMWF ENS forecasts, selected via a clustering selection technique. Forecasts were produced twice per day to 120 hours, with 3-hourly output archived in MARS and disseminated to other COSMO partners. Over the years, COSMO-LEPS has been upgraded on several occasions. Today, it runs a 20-member ensemble producing forecasts to 132 hours.

Following COSMO-LEPS, other Member States started to run either regional or global ensemble numerical weather prediction (NWP) forecasts as TC-2 applications.

The UK Met Office's MOTHS (Met Office Thorpex Suite, later renamed to MOGREPS-15) suite ran as a TC-2 application from 2007 to 2016. It produced a global 15-day, 24-member ensemble forecast based on the Unified Model, primarily as the UK Met Office's contribution to the Thorpex Interactive Grand Global Ensemble (TIGGE) archive.
The Grand Limited-Area Model EPS (GLAMEPS – see Iversen et al., 2020) was a multi-model ensemble prediction system on a limited, pan-European domain which ran as a TC-2 application on behalf of the HIRLAM and ALADIN consortia from 2011 to 2019. It aimed to provide reliable forecasts for the next two to three days with a higher spatial resolution than ECMWF's global ENS forecast at the time.
In 2011, the Austrian national meteorological and geophysical service (ZAMG – now part of the GeoSphere Austria) started to run the ALADIN-Limited Area Ensemble Forecasting (LAEF) suite as TC-2. LAEF was based on the ALADIN/ALARO NWP model and was developed in the framework of the Regional Cooperation for Limited Area modelling in Central Europe (RC-LACE). In 2019, LAEF was replaced by C-LAEF (Convection-permitting LAEF), which still runs today. Output is made available to countries in the LACE community and also provides uncertainties for the Ensemble INCA nowcasting system at ZAMG.
The German National Meteorological Service (Deutscher Wetterdienst – DWD) also ran its Boundary Condition Ensemble Prediction System - Multi Model (BCEPS MuMo) suite as TC-2 between May 2011 and March 2017. The suite interpolated global model data from ECMWF's IFS and forecasts from several national meteorological services to provide LBCs for DWD's COSMO-DE members.

TC-2 applications providing backups to operational NWP activities

Other Member States have used option 2 of the TC Framework to run backups of their operational NWP. For example:

The Instituto Português do Mar e da Atmosfera (IPMA) ran a backup of its ALADIN-Portugal LAM as a TC-2 application between November 2013 and May 2019.
Between January 2015 and the end of October 2021, DWD ran a backup of part of its NWP activity as a TC-2 application in case of a catastrophic disaster at its main site in Offenbach.
In May 2015, Spain's Agencia Estatal de Meteorología (AEMET) began running a backup of its operational HARMONIE suite as a TC-2 application. The model is run for two domains: the Iberian Peninsula and the Canary Islands.
Since July 2019, the National Centre for Aerospace Meteorology and Climatology (CNMCA) of the Italian Air Force Meteorological Service (ITAF) has been using four TC-2 suites to run its NWP system, mostly for back-up purposes. The COSMO model and the NETTUNO ocean wave model are run for two domains and the ICON model for a single domain to provide both deterministic and probabilistic forecasts. A fourth suite runs the KENDA-LETKF atmospheric ensemble data assimilation system to provide initial conditions for the COSMO and ICON forecasts.

Other NWP activities run as TC-2 applications

In more recent years, several Member States have begun to use the TC Framework to run part or all of their operational NWP.

In June 2017, the Irish Meteorological Service, Met Éireann, started to run its full operational NWP production as a TC-2 application. The original deterministic forecasts were based on the HIRLAM and HARMONIE models. By June 2019, these had been replaced by a new HARMONIE-based control plus ten-member ensemble system named IREPS. Nowadays, Met Éireann takes its main NWP products from the United Weather Centres – West (UWC-W) forecasts. However, a skeleton version of IREPS still runs as TC-2.
Met Éireann also elected to run its operational Tidal and Storm Surge Forecasts (TSSF) under the TC Framework. In October 2022, TSSF was the first new suite approved to run as TC-2 on ECMWF's Atos HPC.
With a lack of computer resources preventing an increase of model resolution or the production of ensemble forecasts, the Republic Hydrometeorological Service of Serbia (RHMSS) developed a suite to run its operational forecast model. The suite was approved as a TC-2 application in early 2018.
Between May 2018 and October 2022, the Hellenic National Meteorological Service (HNMS) ran the COSMO model as a TC-2 application to provide high-resolution products for its various operations. Forecasts were run twice per day for two different domains over the Mediterranean Sea and Greece.
In July 2020, A-LAEF (ALARO-LAEF), the RC-LACE consortium's common ensemble prediction system based on the ALARO LAM (see Belluš et al., 2022, for details), began running as a TC-2 application using resources provided by Croatia, Slovenia and Türkiye. Products are distributed to the LACE partners as well as Türkiye, while Poland receives data via servers at the Slovak Hydrometeorological Institute (SHMU).
Since May 2022, the Belgian Meteorological Service (IRM/KMI) has run its operational NWP as a TC-2 application as three different suites: ALARO forced by ARPEGE data, uploaded by IRM to the ECMWF Atos HPCF, is run at 4 km resolution; ALARO and AROME are both run at 1.3 km resolution with initial conditions and LBCs based on IFS forecasts.
Finally, in August 2024, two new TC-2 suites were approved for ESTEA, the meteorological service for Estonia. The suites provide wave forecasts in real time for a domain covering the Baltic Sea in the vicinity of Estonia. One suite makes daily runs of the NEMO-EST ocean forecast model, based on the NEMO-Nordic version of the widely used NEMO ocean model, while the other runs Delft University of Technology's SWAN ocean wave forecast model twice a day.

TC-2 applications processing ENS data

The UK Met Office has also used the TC Framework to run applications which post-process large volumes of ECMWF ensemble forecast data without the need to transfer these first to its Exeter headquarters.

The Site-Specific Post-processing Suite (SSPS) has been running as a TC-2 application since 2013. SSPS extracts site-specific information from ECMWF forecasts for locations around the globe. This is blended with output from UK Met Office models in its operational post-processing system to produce site-specific forecast data.

The UK Met Office has also implemented its Standard Gridding Engine (StaGE) as a TC-2 application which was accepted as a TC-2 suite in September 2024. StaGE processes ECMWF ENS forecast data and delivers products to the UK Met Office in a NetCDF format, ensuring they are in a standard gridded format for use in downstream UK Met Office systems. The primary goal is to make it easier to integrate ECMWF ENS output with that from the Unified Model to produce novel post-processed products.

Israel's activities under TC-2

Although not an ECMWF Member State, Israel has also been able to take advantage of the TC Framework to run operational weather forecasts as TC-2 applications (see Khain et al., 2022). This is thanks to the Optional Programme, which, since 1 January 2020, has allowed Co-operating States without the possibility to become Member States to use ECMWF's HPCF. The Israel Meteorological Service (IMS) has used resources provided under the Programme to run a computationally demanding COSMO ensemble system over the Eastern Mediterranean as well as a high-resolution ICON model over southeastern Europe. These provide weather forecasts for IMS and other weather services in the region. The ICON forecasts are also used within the World Meteorological Organization SEE-MHEWS-A (South-East European Multi-Hazard Early Warning Advisory System) project. In 2023, a third suite for running the ICON model, with initial conditions and LBCs from DWD, was accepted as TC-2, and this year a fourth suite to run an ensemble forecast based on the ICON model was approved.

B

Examples from Member States

Trond Iversen, MET Norway: GLAMEPS (Grand Limited-Area Model Ensemble Prediction System) was a multi-model ensemble numerical weather prediction system on a limited, pan-European domain. The system was run as a time-critical facility (TC-2) at ECMWF from 2011 to 2019. GLAMEPS was technically ambitious and would hardly have been realised without the availability of TC-2. It was a challenge to obtain timely and frequent forecast updates without interfering with ECMWF’s production, which was why six-hourly time-lagging was implemented in an upgrade in 2014. ECMWF staff were crucial for the operational implementation, including a hierarchy of real-time emergency procedures.

Andrea Montani, formerly Arpae-SIMC, now ECMWF: When COSMO-LEPS started to run at ECMWF in November 2002, no single country had the computing capabilities to run the application in a timely manner without interfering with routine operational activities. The cooperation and special efforts provided by ECMWF for the full implementation of the project under the time- critical framework had (and still has!) to be explicitly praised.

Time-critical option 3

Option 3 of the time-critical Framework (TC-3) is an enhancement of TC-2, where ECMWF both monitors and manages the suites. The TC-3 suites are closely integrated with ECMWF's operational suites. They are managed by the Integration Team of ECMWF's Production Section, who also undertake any testing of the suites whenever the IFS is upgraded.

One of the main applications being run as TC-3 provides initial conditions and LBCs based on ECMWF forecasts as Format Arpege files for the AROME, ALARO and ALADIN LAMs, as well as for the MOCAGE chemical transport model. The processing is based on the '903 configuration' available in IFS/ARPEGE. It was originally developed under a Special Project which started in 2006, led by Météo-France and supported by Hungary. Output is disseminated using the ECMWF Product Dissemination System (ECPDS) to members of the LACE community as well as Météo-France, Türkiye, Morocco and Bulgaria. Belgium and Austria also use the output from this TC-3 activity to initialise their forecasts running as TC-2 applications.

Several other TC-3 applications are being run by Météo-France, each of which was developed and run initially as a TC-1 job.

The PERLE (Modèle de dispersion de polluants à échelle locale et régionale) suite runs on demand. It extracts 3D data from MARS on a domain and for a period of time included in a request file, which is sent to ECMWF via ECPDS. The data extracted are sent back to Météo-France, providing part of the input data used to run the PERLE dispersion model. The PERLE suite is triggered typically when there is a chemical incident, or possibly a volcanic eruption, anywhere around the French territories or elsewhere in the world.
ASOT2MIFS (Adaptation Statistique à partir d'Observations de la Température à 2M pour IFS) produces three-hourly and extreme temperature forecasts from statistical adaptations of ECMWF forecast output.
The ALPHA (Algorithmes et modèLes pour la Production Homogène globAle) suite post-processes ECMWF ensemble forecasts using an algorithm developed at Météo-France to create bespoke probabilistic diagnostics.
FAME (Forecast of Airport Meteorological Elements) uses ECMWF forecast output to produce forecasts of variables of interest to the aviation industry at a list of airport locations worldwide.

Other applications currently running as TC-3 are:

Jason2, which post-processes ECMWF analysis fields for EUMETSAT and the Centre national d'études spatiales (CNES - the French national space agency), and
the UK Met Office's extratropical Cyclone Database (CDB) application. This application creates a web-based product that aims to represent, objectively and in a variety of ways, the location and behaviour of near-surface synoptic scale features in ECMWF forecasts. The features represented are those typically associated with adverse weather, such as warm and cold fronts, barotropic lows and frontal and diminutive waves.

The currently running TC-3 suites viewed in ecFlowUI are shown in Figure 4.

**FIGURE 4** The TC-3 suites active in November 2024, viewed in ecFlowUI.

Using the TC Framework in emergency situations

On two occasions, the TC Framework has been used to support ECMWF Member States in emergency situations.

First, in August 2012, the Danish Meteorology Institute (DMI) experienced a series of problems with its power supply, resulting in severe damage to some of the central components of its HPCF, which consequently went out of service. DMI was already running a version of its HIRLAM model on the ECMWF supercomputer. It contacted ECMWF to ask if it would be possible to get high-priority access to sufficient HPC resources for a few days so that its NWP output could continue to be produced in a timely fashion while engineers worked on repairs to its HPCF. Access was quickly enabled to allow DMI to run in the higher-priority batch queues that are reserved for time-critical work under the TC Framework.

On a second occasion, a 5.3 magnitude earthquake on Sunday, 22 March 2020 severely damaged the headquarters of the Croatian Meteorological and Hydrological Service (DHMZ) in Zagreb. DHMZ quickly ported its NWP activities and some post-processing tasks to ECMWF's HPCF (see Abellan et al., 2020). Within a matter of days, the main components of the NWP system were running on the Centre's supercomputers, again making use of the higher-priority batch queues provided through the TC Framework to enable the forecasts to run in a timely manner.

Current status and future outlook

Twenty years after the TC Framework was approved, it remains very popular with Member and Co-operating States. Four new TC-2 suites were approved during 2024, bringing the total to 23 suites being run by 11 different countries and consortia. Two further suites are being prepared and are expected to be approved as TC-2 within the next 12 months. There is also a major upgrade planned for the oldest TC-2 application, COSMO-LEPS, with the COSMO model being replaced by ICON.

There have also been three new TC-3 suites added during the last 12 months to provide initial conditions and LBCs as Format Arpege files for Morocco, Türkiye and Bulgaria. This brings the total number of TC-3 suites being run to 17 on behalf of 11 countries, organisations and consortia. For the future, Météo-France is preparing two further potential TC-3 activities to support forecasts for commercial aviation. The TC-2 and TC-3 suites active in November 2024 are depicted schematically in Figure 5.

**FIGURE 5** TC-2 (in blue boxes) and TC-3 (in green boxes) suites at the beginning of November 2024, grouped by country, organisation or consortia.

The TC-1 activity also remains very popular with Member and Co-operating State users. During August 2024, around 150 different users ran a total of almost 55,500 jobs (about 1,800 jobs per day on average), with each job subscribed to one of 102 different defined 'events'. In 2024, the Aviso service became operational (Iacopino et al., 2021). This provides users on the ECMWF HPCF or the European Weather Cloud with notifications when disseminated data or data in MARS are available, allowing them to trigger workflows at remote sites. However, Aviso is not a direct replacement of TC-1 events as it does not provide a monitoring facility or a re-run of failed jobs by ECMWF.

Recently a review of the TC Framework was undertaken to understand where updates and improvements can be made. The review took into consideration best practices and lessons learned from experiences with ECMWF's operational suites and with Member States' time-critical activities. It also looked at new technologies, such as the Atos HPCF, the European Weather Cloud, containerised workflows, and dedicated Virtual Machines for hosting the ecFlow servers. The review generated several straightforward and non-controversial suggestions for improving the Framework, which are gradually being implemented. Other recommendations, such as the possibility of using the European Weather Cloud to run certain applications, are still being explored.

Over the years, the TC Framework has enabled ECMWF Member States to run workflows which they could not easily run themselves. These include running high-resolution LAMs or ensemble forecasts or post-processing large volumes of data to produce novel products, in a routine and timely manner. Clearly, the TC Framework continues to provide great benefit to ECMWF's Member and Co-operating States, and we look forward to the next 20 years of the service.

Editorial

News

Earth system science

Computing

Newsletter