Global Surface Archives

Global Surface Archives was introduced by Weather Graphics in 2011 and constitutes the single largest collection of observed hourly and synoptic weather data in existence. You don't get cryptic XML or JSON records, or worse yet, a packed binary file. All data is in plaintext and available in SYNOP, METAR, or CSV format. These standard formats can be browsed in text editors by most meteorologists and can be plotted in a variety of off-the-shelf software tools.

For detailed historical maps, researchers traditionally had to rely on the NCEP Reanalysis series, which is a binary dataset requiring specialized viewers and is mostly limited to 250 km resolution. In Global Surface Archives, you have all of the actual observations and can construct even the most demanding mesoscale analyses for years gone by, whether you're looking at the 1999 Moore Oklahoma tornado or the 1984 Ivanovo Russia tornado. Global Surface Archives represents so much processing work that it took 1 month of continuous processing time to complete on an Intel Core i7 machine.


ORDERING

Can't decide? You may order the Standard Version and upgrade later for the difference in price.

Standard Version. This version includes Library 1 (METAR/SYNOP, one file per date/hour; see margin for details).
GSS001 | Blu-Ray ($240) | DVD ($240)

Professional Version. This version includes all libraries: 1, 2, and 3 (see margin for details).
GSP001 | Blu-Ray ($335) | Not distributed on DVD

Lifetime Updates. Lifetime Updates is a VIP service that allows you to request updated discs at any future date as long as we're maintaining this product line. Updates are in January and sometimes in July. Shipping is free. You don't need to add this at the time of purchase; you may add it at a later date.
GSL001 | Buy ($95)


MAPS PLOTTED WITH GLOBAL SURFACE ARCHIVES

The Chernobyl disaster is depicted on this map valid 1200 UTC on 26 April 1986 (click for large version). The dataset shows that all of Ukraine under the influence of a 1034 mb anticyclone over the White Sea, which was producing easterly wind components across far western Russia, Ukraine, and Belarus. This pattern was partly responsible for advecting some of the contamination northwestward into the Baltic region, contributing to widespread alarm throughout western Europe.
A record heat wave in England (28 June 1976, 1800 UTC) (click for large version) which brought temperatures well above 30°C throughout much of England. Temperatures reached 35.6°C (96.1°F) in Southampton, the highest June temperature recorded in the UK. Unprecedented drought conditions led Parliament to pass the Drought Act, giving local authorities the power to control the use of water in their jurisdictions. The northeasterly flow into Great Britain is somewhat ironic since it is this pattern which, in the winter, is associated with cold waves. However, in this instance the trajectory off the European continent is bringing warm, dry air rather than cold, dry air.
The lowest temperature to ever occur on Earth (21 July 1983, 0000 UTC) (click for large version) occurred at Vostok, Antarctica. There's not much to see on this map because the station density in Antarctica has always been historically sparse, but there it is. Vostok had started out at -60°C (-76°F) on July 7th, then two weeks of clear nights in the "perpetual night" of the Antarctic winter allowed heat to radiate away before the site finally reached -89.2°C (-128.5°F) as shown here. That's a 16-knot wind. We tried to calculate wind chill with the NWS/SRH wind chill calculator, but at those readings it just says "Undefined".
TECHNICAL ASPECTS


Contents
Global Surface Archives is a ultra-massive collection of worldwide "hourly" and "synoptic" weather observations in two of the most universal formats for weather data distribution. The size of this dataset is approximately 39 GB (170 GB uncompressed).

Period of record
Datasets start in 1930, though a few records exist for Finland and France going back to 1920 and were included. The data is sufficiently dense to allow synoptic maps of the US in the late 1930s and for parts of the US and Europe in the 1940s. Strong station density emerges worldwide in the 1950s.

Library sets
One size doesn't fit all, so we offer Global Surface Archives in three flavors, or "libraries". Different libraries contain the exact same information as other libraries, but the ordering or formatting is different. The Standard Edition gives you Library 1, which is adequate for most casual use and is similar to the Archives products we've sold over the years. The Professional Edition gives you Library 1, 2, and 3.

  • Library 1. Observations are in SYNOP and METAR format. Each file contains data for a unique date/time. A filename might appear as 1995080214_metar.txt, containing all worldwide observations for August 2, 1995 at 1400 UTC. This allows easy plotting by weather mapping software like Digital Atmosphere. This library is the one most requested by most of our customers and is very similar to the various Archives sets we have sold for the past 15 years. Click here for a sample of Library 1 data.
  • Library 2. Observations are in SYNOP and METAR format. Each file contains data for a unique station. The station files are also split into years, since the row count would otherwise overload most text editors and word processors. A filename might appear as FAJS-1986.txt, which would contain all observations for Johannesburg, South Africa in 1986. Click here for a sample of Library 2 data.
  • Library 3. Observations are in tabular (CSV) format. Each file contains data for a unique station. The CSV format is decoded and can be easily opened in Excel or OpenOffice. The station files are also split into years, since the row count would otherwise overload most spreadsheet programs. A filename might appear as RJTT-1997.csv, which would contain all observations for Haneda Airport in Tokyo for 1997. Library 3 contains land stations only (no ships). Click here for a sample of Library 3 data.

Identifiers
METAR data is indexed according to standard ICAO codes. To enhance the usability of this dataset, we use a version of this called Epoch 2010, in which only identifier assignments which were valid in the year 2010 are used. For example, a 1975 observation from Hanoi Gia Lam would appear as VVGL, same as the current identifier. We wouldn't use VWHN, which is what the pre-1976 North Vietnamese government used. Doing so would require all our users to consult old ICAO documents to see what the identifier was used for. We've taken care of all the hard work for you. SYNOP data is indexed according to standard 5-digit World Meteorological Organization station numbers; since these do not change they are provided in their original assignment as received.

WMO Restrictions
SYNOP reports for about 40 countries at intermediate hours (outside of 00, 06, 12, and 18Z) falls under the purview of WMO Resolution 40 and is omitted from this dataset. For more information about Resolution 40 see the WMO page on the topic. Where data is omitted, METAR data tends to be sufficiently abundant to cover the shortfall.

Known deficiencies
There are a number issues we are aware of and because of this, the dataset is sold as-is. Worldwide data is very scarce during the 1930s and somewhat scarce during the 1940s since those were the primitive years of hourly observations. It may be impractical to construct satisfactory maps until the 1950s. Also, much of the data is reassembled from "packed" records; before the 1980s mass storage space was extremely expensive and archiving data in original form was considered wasteful. All METAR data is also standardized to the post-1996 standard, which is generally considered to be a bonus but does cause some degradation of sky condition data, which was encoded in a different system before then. There is a shortfall in observations between 1969 and 1972 due to data loss at the telecommunications centers and results in data not being available in some countries; the data loss episode peaked at 1972 before recovering in 1973.

Inventory
A basic inventory of the dataset is provided here. These ZIPs contain a large text file that lists every file in the set along with its size.
lib001.zip - Library 1 (METAR/SYNOP by date/time)
lib002.zip - Library 2 (METAR/SYNOP by station)
lib003.zip - Library 3 (tables by station)

Structure
The disc structure is described as follows:

LIBRARY 1 (DVD version)
Disc Contents Uncompressed data Compressed data
Disc 1 1920-1971 15.5 GB 4.0 GB
Disc 2 1972-1980 16.1 GB 3.9 GB
Disc 3 1981-1987 15.6 GB 3.8 GB
Disc 4 1988-1993 15.2 GB 3.7 GB
Disc 5 1994-1999 17.7 GB 4.3 GB
Disc 6 2000-2002 14.8 GB 3.4 GB
Disc 7 2003-JUN05 17.0 GB 3.7 GB
Disc 8 JUL05-DEC07 21.0 GB 4.3 GB
Disc 9 JAN08-JAN10 16.8 GB 3.7 GB
Disc 10 FEB10-DEC11 20.2 GB 4.3 GB
TOTAL TOTAL 169.9 GB 39.2 GB



LIBRARY 1 (Blu-Ray version)
Disc Contents Uncompressed data Compressed data
Disc 1 1920-2001 89.3 GB 21.9 GB
Disc 2 2002-2011 88.0 GB 19.4 GB
TOTAL TOTAL 177.2 GB 41.3 GB



LIBRARY 2 (Blu-Ray only)
Disc Contents Uncompressed data Compressed data
Disc 1 SYNOP / METAR A-P 138.1 GB 22.9 GB
Disc 2 METAR Q-Z 24.2 GB 3.7 GB
TOTAL TOTAL 162.3 GB 26.7 GB



LIBRARY 3 (Blu-Ray only)
Disc Contents Uncompressed data Compressed data
Disc 1 SYNOP / METAR A-P 195.2 GB 22.9 GB
Disc 2 METAR Q-Z 23.3 GB 2.6 GB
TOTAL TOTAL 218.6 GB 25.4 GB