Newsletter-banner-No-150

Twenty-one years of wave forecast verification

Jean-Raymond Bidlot

 

Routine comparisons of wave forecast data from different models were first informally established in 1995. They were intended to provide a mechanism for assessing the quality of operational wave forecast model output. The comparisons were based on an exchange of model analysis and forecast data at the locations of in-situ observations of significant wave height, wave period and wind speed and direction available via the Global Telecommunication System (GTS). Five European and North American institutions routinely running wave forecast models contributed to that exchange (Bidlot et al., 1998). The Expert Team on Wind Waves and Storm Surges of the Joint Technical Commission for Oceanography and Marine Meteorology (JCOMM) noted the value of the exchange during its first meeting in Halifax, Canada, in June 2003 and endorsed the expansion of the scheme to include other wave forecasting systems. The exchange was subsequently expanded to other global wave forecasting centres and a few regional entities (Table 1).

Table 1: Current contributors to the wave forecast verification project. The start date indicates the date from which data have been provided. Data coverage is either global or regional. The number of forecasts per day and the forecast range refer to the data that is transmitted for verification purposes and not to what each centre provides to its users.
 

Organisation

Acronym

Start date

Coverage

Forecasts
per day

Forecast range (days)

1

European Centre for Medium-Range Weather Forecasts, UK

ECMWF

Jun 1995

global

2

10

2

Met Office, UK

UKMO

Jun 1995

global

2

5

3

Fleet Numerical Meteorology and Oceanography Center, USA

FNMOC

Jun 1995

global

4

6

4

Environment and Climate Change Canada, Canada

ECCC

Jun 1995

regional until June 2015, then global

2

5

5

National Centers for Environmental Prediction, USA

NCEP

May 1996

global

4

7

6

Météo France, France

METFR

Jan 2001

global

2

5

7

Deutscher Wetterdienst, Germany

DWD

Feb 2004

global

2

5

8

Bureau of Meteorology, Australia

BoM

Sep 2005

global

2

5

9

Service Hydrographique et Océanographique de la Marine, France

SHOM

Sep 2006

global

2

6

10

Japan Meteorological Agency, Japan

JMA

Sep 2006

global

4/1

3.5/10

11

Korea Meteorological Administration, Republic
of Korea

KMA

Jan 2007

global

2

10

12

Puertos del Estado, Spain

PRTOS

Jan 2007

regional

2

3

13

Danmarks Meteorologiske Institut, Denmark

DMI

Jan 2010

regional

4

5

14

National Institute of Water and Atmospheric Research, New Zealand

NIWA

Jun 2010

global

1

6

15

Det Norske Meteorologiske Institutt, Norway

METNO

Feb 2011

regional

4

2

16

Servicio de Hidrografía Naval, Servicio Meteorológico, Argentina

SHNSM

Aug 2011

regional

2

4

A review of 21 years of wave verification results shows clear improvements in the quality of wave forecasting, as will be illustrated in this article for significant wave height forecasts. The comparison project has benefitted all participants and should continue to do so. However, the informal character of the exchange prevents a rapid adaptation to new data. For these reasons, the World Meteorological Organization (WMO) is seeking to establish a Lead Centre for Wave Forecast Verification (LC-WFV) with clearly defined interfaces between the participants and the Lead Centre. ECMWF has expressed its interest in becoming the designated Lead Centre.

Data

On a monthly basis, each participating centre provides time series of model data at an agreed list of locations to ECMWF, where the data are collated for subsequent access. Observations are also collected at ECMWF. The combined data are then processed to provide summary statistics. These are made available on the ECMWF website (http://www.ecmwf.int/en/forecasts/charts/) and the JCOMM website (http://www.jcomm.info/). The raw data are also made available to all participants for potential further analysis.

Sea state and ocean surface meteorological in-situ observations are routinely collected by several national organisations via networks of moored buoys or weather ships and fixed platforms deployed in their near-shore and offshore areas of interest. The data are usually exchanged via the GTS. As part of this intercomparison, observations that are not commonly available on the GTS are also gathered on a case-by-case basis. The geographical coverage of the wave data is still very limited. It tends to be limited to areas near the coast and some observations are very close to land. At the present global wave model resolution, only a subset of these locations fall within the wave model grids. Most measurements used in this project are made in the northern hemisphere (see Figure 4 for recent coverage).

Before using observations for verification, care has to be taken to process the data to remove any erroneous observations. It is also necessary to match the temporal scale of model data and observations. This scale matching is achieved by averaging the hourly observation data in time windows centred on verifying times. The original quality control and averaging procedure was discussed in Bidlot et al. (2002). It was extended to include platform data as described in Sætra & Bidlot (2004).

The intercomparison relies on the exchange of model output at a list of locations. Because in-situ networks change over time, updates to the list have been necessary. However, not all participants have been able to update their list at the same time, nor do they provide data for all the same locations. Moreover, some participants only run a limited-area model, use a coarser grid or provide data from a different number of forecasts (Table 1). A fair comparison between the different wave forecasting systems can only be achieved if the same observation–model collocations are used. This constrains the number of systems that can be evaluated at any one time.

Over 20 years of progress

A

Scatter index

The scatter index is a measure of the size of the deviation of forecasts from observations relative to the magnitude of the observations. It is normally given in per cent. A smaller scatter index value means better forecasts.

Mathematically the scatter index is defined as the standard deviation of the difference between predicted values and observations normalised by the mean of the observations. For example, if the standard deviation of the difference between predicted values of significant wave height and observations is 0.5 metres and the mean of the observations is 2 metres, then the scatter index value is 0.5/2, which is 25%.

Significant wave height is defined as four times the square root of the integral of the wave spectrum. It closely corresponds to the average height of the highest one third of waves.

Figure 1a shows the significant wave height forecast skill from September 2015 to August 2016 as measured by the scatter index (Box A) for all systems providing global forecasts from 00 UTC (see Table 1). Figure 1b shows the common locations and the data coverage density (the number of observation-model collocations used relative to the maximum number of possible collocations). This article does not aim to explain why each forecasting system performs differently. Rather, it aims to illustrate the remarkable progress that has been made over the years (Figures 2 and 3). Progress might have come from improvements in atmospheric forcing resulting from a collective effort in developing numerical weather prediction (NWP) systems, and/or advances in the wave model physical parametrizations, numerical methods, data assimilation or improved implementation. It is, however, worth mentioning that METFR and SHOM both use winds from ECMWF, which explains their close similarity with ECMWF in terms of forecast performance.

Figure 1
%3Cstrong%3EFigure%201%3C/strong%3E%20Forecast%20performance%20of%20different%20centres%20for%20forecasts%20initialised%20at%2000%20UTC%20and%2012%20UTC%20between%20September%202015%20and%20August%202016,%20showing%20(a)%20the%20scatter%20index%20(%25)%20for%20significant%20wave%20height%20when%20compared%20to%20buoy%20observations%20for%20different%20forecast%20ranges%20and%20(b)%20the%20buoy%20positions%20and%20the%20number%20of%20observation-model%20collocations%20used%20relative%20to%20the%20maximum%20number%20of%20possible%20collocations%20over%20this%20one-year%20period.
Figure 1 Forecast performance of different centres for forecasts initialised at 00 UTC and 12 UTC between September 2015 and August 2016, showing (a) the scatter index (%) for significant wave height when compared to buoy observations for different forecast ranges and (b) the buoy positions and the number of observation-model collocations used relative to the maximum number of possible collocations over this one-year period.

Figure 2a shows the evolution of 5-year running mean scatter index values for day-5 significant wave height forecasts for an area of the North-East Pacific. The selected offshore buoys have been part of the intercomparison since the early years. The plot was produced with consistent 00 and 12 UTC forecasts at all selected locations. The data coverage density over the full period is also shown (Figure 2b). It is not entirely uniform but the locations have been carefully selected to reflect the wave climate of the area. The decrease in scatter index values is a clear indication of the steady improvements made by all participating centres. There is a degree of convergence in model performance since 2009. Comparable results also hold for shorter forecast ranges (not shown). Similarly, other ocean areas with long-term observational coverage, such as the North-West Atlantic, the North-East Atlantic and the North Sea, generally show the same improving trend for all participants and forecast ranges (Figure 3a–c). However, for enclosed areas such as the Western Mediterranean Sea, progress has been less consistent (Figure 3d).

Figure 2
%3Cstrong%3EFigure%202%3C/strong%3E%20Forecast%20performance%20of%20different%20centres%20for%20forecasts%20initialised%20at%2000%20UTC%20and%2012%20UTC%20showing%20(a)%20the%20long-term%20evolution%20of%205-year%20running%20mean%20scatter%20index%20values%20for%20day-5%20significant%20wave%20height%20forecasts%20when%20compared%20to%20buoy%20observations%20over%20the%20North-%20East%20Pacific%20and%20(b)%20the%20buoy%20positions%20and%20the%20number%20of%20observation%E2%80%93model%20collocations%20used%20relative%20to%20the%20maximum%20number%20of%20possible%20collocations%20over%20the%2021-year%20period.
Figure 2 Forecast performance of different centres for forecasts initialised at 00 UTC and 12 UTC showing (a) the long-term evolution of 5-year running mean scatter index values for day-5 significant wave height forecasts when compared to buoy observations over the North- East Pacific and (b) the buoy positions and the number of observation–model collocations used relative to the maximum number of possible collocations over the 21-year period.

Figure 3
%3Cstrong%3EFigure%203%3C/strong%3E%20Same%20as%20Figure%202%20but%20for%20(a)%20day-1%20forecasts%20for%20the%20North-West%20Atlantic,%20(b)%20day-3%20forecasts%20for%20the%20North-East%20Atlantic,%20(c)%20day-1%20forecasts%20for%20the%20North%20Sea,%20and%20(d)%20day-1%20forecasts%20for%20the%20Western%20Mediterranean%20Sea,%20for%20different%20sets%20of%20forecasting%20centres.
Figure 3 Same as Figure 2 but for (a) day-1 forecasts for the North-West Atlantic, (b) day-3 forecasts for the North-East Atlantic, (c) day-1 forecasts for the North Sea, and (d) day-1 forecasts for the Western Mediterranean Sea, for different sets of forecasting centres.

Figure 4
%3Cstrong%3EFigure%204%3C/strong%3E%20Scatter%20index%20for%20day-1%20ECMWF%20forecasts%20of%20significant%20wave%20height%20initialised%20at%2000%20and%2012%20UTC%20every%20day%20from%20September%202015%20to%20August%202016%20compared%20to%20buoy%20observations,%20shown%20for%20each%20buoy%20location.
Figure 4 Scatter index for day-1 ECMWF forecasts of significant wave height initialised at 00 and 12 UTC every day from September 2015 to August 2016 compared to buoy observations, shown for each buoy location.

ECMWF data can be collocated with all available in-situ data. Figure 4 shows that enclosed areas and near-shore locations are indeed much more difficult to model, in particular on the western side of all ocean basins. This is not limited to ECMWF but is a feature of forecasts from all centres (Figure 5). Nonetheless, the quality of wave forecasting as a whole has improved quite dramatically. There is obviously room for further advances. It is believed that institutions engaged in wave forecasting will continue to benefit from this type of inter-validation in the same way as NWP centres have benefitted from the exchange of forecast verification scores under the auspices of the WMO.

Figure 5
%3Cstrong%3EFigure%205%3C/strong%3E%20The%20top%20left%20panel%20shows%20scatter%20index%20values%20for%20ECMWF%20day-1%20significant%20wave%20height%20forecasts%20as%20in%20Figure%204.%20The%20other%20panels%20show%20the%20difference%20in%20scatter%20index%20expressed%20in%20per%20cent%20with%20respect%20to%20ECMWF%20at%20each%20location%20from%20other%20participating%20centres%20to%20the%20extent%20that%20data%20were%20available%20from%20September%202015%20to%20August%202016.
Figure 5 The top left panel shows scatter index values for ECMWF day-1 significant wave height forecasts as in Figure 4. The other panels show the difference in scatter index expressed in per cent with respect to ECMWF at each location from other participating centres to the extent that data were available from September 2015 to August 2016.

Outlook

There has been a slow, yet steady increase in the availability of in-situ wave observations. Space-borne altimeter wave height data have been shown to be of very high quality and are now commonly available (Abdalla & Zuo, 2016). The intercomparison should ideally be extended to include these data. The JCOMM Expert Team on Waves and Coastal Hazards has recommended that the current Wave Forecast Verification project should be formalised by establishing a Lead Centre for Wave Forecast Verification (LC-WFV). ECMWF has responded positively to this request. The designated LC-WFV would coordinate efforts to gather a set of selected model fields relevant to wave forecasting activities under an agreed data exchange protocol. Once the process of gathering the relevant fields is in place, the routine verification against in-situ data will be more flexible and adaptive. Moreover, it will become much easier to include new observational datasets and verification metrics.

 

The author would like to thank Andy Saulter (UK Met Office), Paul Wittmann (FNMOC), Natacha Bernier (ECCC), Arun Chawla (NCEP), Lotfi Aouf (Météo-France), Thomas Bruns (DWD), Aihong Zhong (BoM), Fabrice Ardhuin (SHOM), Nadao Kohno (JMA), Sanwook Park (KMA), José María García-Valdecasas Bernal (PRTOS), Jacob Woge Nielsen (DMI), Richard Gorman (NIWA), Ana Carrasco (METNO) and Paula Etala (SHNSM) for their contribution to the comparison project and for providing the data that has made this article possible.

 

Further Reading

Abdalla, S. & H. Zuo, 2016: The use of radar altimeter products at ECMWF. ECMWF Newsletter No. 149, 14–19.

Bidlot, J.-R., M. Holt, P.A. Wittmann, R. Lalbeharry & H.S. Chen, 1998: Towards a systematic verification of operational wave models. Proceedings Third Int. Symposium on WAVES97: November 3-7, 1997, Virginia Beach: American Society of Civil Engineers.

Bidlot, J.-R., D.J. Holmes, P.A. Wittmann, R. Lalbeharry & H.S. Chen, 2002: Intercomparison of the performance of operational ocean wave forecasting systems with buoy data. Wea. Forecasting, 17, 287–310.

Sætra, Ø. & J.-R. Bidlot, 2004: On the potential benefits of using probabilistic forecasts for waves and marine winds based on the ECMWF ensemble prediction system. Wea. Forecasting, 19, 673–689.