The climate reanalysis data challenge

Looking at how weather patterns have changed over the decades can help us gain insights into climate change. This requires observations going as far back as possible, but finding such data is a challenge. At the same time, advances in science, for example using state-of-the-art reanalysis, mean that more information can be extracted from past data than when the observations were made.

Reanalysing the 20th century

In 2014, working with partners across Europe, ECMWF produced its first atmospheric reanalysis of the entire 20th century, ERA-20C. Reanalysis combines the latest science with as many observations as possible to reconstruct the past climate. Its results are important for validating climate models, improving reference climate datasets for numerical weather prediction, and providing a baseline to place today’s weather events into the climate context.

Valuable information for climate change services

Recovering older observations from meteorological organisations, satellite agencies, universities and libraries was a key part of the EU-funded ERA-CLIM project that produced ERA-20C. ERA-CLIM2 continues the work to improve access to climate data, with an emphasis on extending the usable satellite climate data records. “People don’t realise that some of the old satellite data from the 1970s, which have been residing on magnetic tapes, actually contain valuable information on climate change for some essential climate variables,” says Dr Roger Saunders from the UK Met Office. Reanalysis data will contribute to the Climate Data Store at the heart of the new Copernicus services.

“Ageing media”

Over the last 50 years, satellite agencies have gathered enormous volumes of data about our environment. Each mission pushed the limits of data transmission, processing, and archiving capacity. The speed and capacity of archiving media at the time  – top-of-the-range then – pale in comparison with the capabilities of today’s hand-held smartphones. The difficulty of accessing historical satellite data has hampered their use until recently.

The geostationary data from NOAA over the US is a case in point. Dr David Santek from the University of Wisconsin-Madison Space Science and Engineering Center (SSEC) explains: “We are nearly done recovering historical images from 1979 to 1996 from original tapes to new media, and that is timely for three reasons: these tapes were deteriorating beyond use; today’s technology allows us to eventually serve the data online; and the only colleague who knows how to work the old tape readers is nearing retirement. Next on our list are 9-track tapes containing Geostationary Operational Environmental Satellite (GOES) data from 1975 to 1979. It is not a complete record, but we're attempting to read these tapes now.”

Dr Paul Poli from ECMWF’s Reanalysis Section adds, “this is also about raising awareness, so that we do not lose any more data; even at well-established institutions, old physical media, especially the non-conventional ones, tend to ‘disappear’ when organisations move or make room for newer data.”

Examples of precursor instruments whose environmental datasets seem to have been misplaced forever include an ozone spectrometer on the Atmosphere Explorer-E (also known as Explorer 55), flown by the US Air Force, and several infrared vertical temperature and humidity profile radiometers flown by the Defense Meteorological Satellite Program, all in the 1970s and early 1980s. Dr Poli’s search for these data has met with unhappy realisations from counterparts, such as “tapes sadly discarded” or “tapes unfortunately long since been destroyed”. Data from the 1970s can extend the current satellite time series back another decade and Dr Saunders warns “it is vital we preserve and rescue these valuable observations before it is too late.”

Dr Poli stresses the importance of preserving the raw, “level 0” data. Everything downstream of that has been affected by processing, and often there is no good record of exactly what processing took place. This brings about another challenge. “As the pioneers of the space race and satellite meteorology approach retirement, it is critical to find out from them what the data represent,” Dr Poli says. “These people have essential mission and instrument knowledge – nowadays called metadata – that may have never been published elsewhere.”

Unique data

With near-global coverage, satellites often provide information unavailable from any other source. As NASA reported in 2014, early satellite data has much to add to what scientists already know about past weather. ("Nimbus: NASA Remembers First Earth Observations")

When it comes to characterising changes in the atmosphere, vertical soundings are required. The first interferometer in space was the IRIS instrument built by NASA in the 1960s. Modified versions of IRIS are still flying today on the Voyager spacecraft, but the IRIS instrument that flew on Nimbus-4 in 1970/1971 collected thousands of spectra emitted by the Earth’s atmosphere and surface. These data, now available from NASA, are truly unique. For example they will help refine polar stratospheric ozone estimates during the polar night in 1970, when the only other instrument observing ozone from space was measuring back-scattered Ultra-Violet daylight.

New tools

Scientists now have more powerful tools to work with the data. Thanks to increased computing capacity, datasets considered very large at the time can now be processed in a few hours. ECMWF’s computing facilities and models make it well placed to realise the value of this information and align it with other environmental records. Key steps include adding ancillary information from various reanalyses (e.g. about the likely weather at the time or the surface type), carrying out quality control, and using the data in modern models and data assimilation systems.

Collaboration enabler of progress

In recent years, various individual researchers, as well as larger-scale programmes such as ERA-CLIM have made notable progress. Space agencies are undertaking their own programmes to secure their heritage data. There are also substantial records waiting to be exploited from the former Soviet Union and associated countries.

Back in 2009, Dr Poli began an inventory of satellite datasets of potential value to global reanalysis. His research triggered interest and became the basis for work with the UK Met Office as part of ERA-CLIM.

The table below shows historical satellite datasets identified as valuable in 2009, now located, rescued, or reprocessed.

The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC), which is working to recover datasets from a number of these older missions, was helpful in adjusting their work queue to make these data available as quickly as possible.

Other efforts of recovery are more grass roots. After an email quest, in 2011 Dr Anthony McNally from ECMWF located copies of four datasets in an English library from instruments designed by the British scientific community. The UK Met Office is now working on these datasets, with support from EUMETSAT’s Satellite Application Facility for Numerical Weather Prediction.

Another example involves data from the microwave instrument SMMR that operated 1978–87. “We had high-level retrievals but were searching for the raw data,” explains Dr Poli. “It was through a network of colleagues asking around that the data were eventually located in the US and passed on to EUMETSAT’s Satellite Application Facility on Climate Monitoring.”

Boost to reanalysis

Reanalysis can benefit in more than one way from satellite records such as those from Nimbus-4 IRIS. That mission lasted about 10 months, making it too short to make, by itself, a substantial difference in a 100-year reanalysis, but piecing together the various missions will help patch the record. A more immediate benefit comes from the high quality and high spectral resolution of Nimbus-4 IRIS data. The reference points they provide help verify the quality of reanalyses for such early years. By doing so, they can help identify the deficiencies in models and data assimilation that need addressing for further progress.

Satellite images from 1970, 2003 and 2014 showing the evolution of the brightness temperature at a wavelength of 9.6 microns in October over the South Pole. This brightness temperature, measured in Kelvin, is sensitive to ozone. When ozone is depleted in the stratosphere, this region of the atmosphere becomes transparent to radiation at this wavelength emitted from underneath by the surface and the clouds, and satellites see warmer temperatures (more red spots in 2003 and 2014). The black spots, more numerous in 1970, represent areas without data. Current sounders such as AIRS and IASI collect many more data thanks to cross-track scanning, a technology unavailable to advanced instruments like IRIS in 1970.

Studies also point to the potential for a better water cycle representation in reanalysis by analysing where the water vapour is located. A recent study supported by EUMETSAT’s Satellite Application Facility on Climate Monitoring has determined that data from the 1990s from SSM/T-2, the first operational microwave water vapour sounder to be placed in orbit, should improve the quality of future reanalysis for the 1990s.

An essential supporting role

Past space missions used the best technology around at the time, and mobilised great resources from the nations that supported those efforts. With most of the work already done, satellite data rescue continues the return on past investments. There are, in addition, greater expectations.

Data recovery programmes are just starting to take root, says Dr Poli: “The next steps are to assimilate the data rescued to extend the reanalysis record further into the past.” These longer records will provide a better basis for assessing changes in the atmosphere using new and more accurate data, in particular from the Copernicus Sentinel missions. By doing so, the old data will act as supporting roles for the “rising star” data that Copernicus is expected to collect.

Photograph courtesy of NASA. "The Thor-Agena-10 launched the Nimbus III, Earth observation and meteorology satellite, on April 13, 1969."