INTERNET MENU Retrieve data

This module retrieves alphanumeric data through the Internet and saves it to your hard drive. When the data download is completely finished, Digital Atmosphere will import the data for you. This means you can immediately begin plotting and analyzing.

To use Retrieve Data, you must first open a connection to the Internet through your service provider before this command will work. Note that you must have a standard Winsock Internet connection. Firewalls or unusual configurations could cause problems. If you have a proxy server, you may want to enter your proxy host information in Preferences > Misc.

When you select Retrieve data, you will be presented with a dialog box containing a list where you can check off the items you want to download (see illustration) from a product list. Once you press Retrieve, the chosen products will be downloaded and stored in your data directory. Then you can use File, Import Directory to see the results.

Since Internet sites often change, you can fully edit the list of products and data directly from this dialog box by highlighting the item you wish to change and clicking the Edit Site button or the Delete Site button. You can also add a new site by clicking the Add Site button. You will then be prompted to enter the site name and URL for the data.

If you begin a URL with %1, such as %1http://www.mysite.com/data/mydat this instructs Digital Atmosphere to search the directory to find the most recent file. In this case it would search http://www.mysite.com/data/ . The filename on the end (in this case "mydat") is used as a wildcard to find matching files (so files like mydatafile.txt and mydata084.dat would be included, while wxdata.dat would be discarded. If you omit the filename, all files in the data directory are included in the search. It is not advisable to use this directory search function with all the products in your Internet table, because with university servers processing METAR and SYNOP data, coding errors may result in the creation of extraneous files that do not contain data. The directory search function is recommended instead for certain NEXRAD and other products on certain servers when there is no way to effectively predict the filename. Also be aware that some servers are unable to present directory listings due to security reasons.

Pointer page search

If you carry a URL with two parts, separated by a space, it indicates that a pointer search will be done.  This is commonly used for dynamically-generated web pages where the target URL is not known.

The first portion is the pointer URL and the second part is the pointer search string, as shown here:
     pointerurl pointersearch

If Digital Atmosphere encounters this, it will go to the URL indicated by "pointerurl", then will search that page for any text fragment containing "pointersearch", obtain the location where this occurs within quotation marks, and add this to the pointer URL to construct a new URL.

For example, this URL:
          http://www.spc.noaa.gov/products/outlook/day1otlk.html /products/outlook/archive/
is used by Digital Atmosphere to obtain convective outlooks.  Obtaining the first page, we open it and find this HTML fragment:
         CLICK TO GET <a href="/products/outlook/archive/2006/KWNSPTSDY1_200612301300.txt">WUUS01 PTSDY1</a> PROD
Digital Atmosphere extracts the "/products/outlook/archive/2006/KWNSPTSDY1_200612301300.txt" part and appends it to the pointer URL to get the final URL of:
         http://www.spc.noaa.gov/products/outlook/archive/2006/KWNSPTSDY1_200612301300.txt

Many sites, especially for surface and upper air data, have a filename that is dependent on the Greenwich Mean time of the observation. To support this, Digital Atmosphere lets you insert time tokens into a filename. For example, if you entered a product that had a URL address of http://www.weatherdata.com/@Hdata.txt, Digital Atmosphere would automatically substitute the current UTC (GMT) hour in place of the @H. So if it was 9 pm EST (02Z), Digital Atmosphere would retrieve http://www.weatherdata.com/02data.txt.

Here is a complete list of all the UTC time token combinations you can use:

Character 1 - mode
@ This hour
$ Last hour

Character 2 - time
M Month
D Day
H Hour
X Rawinsonde hour (rounds down to 00 or 12)
S Main synoptic hour (rounds down to 00, 06, 12, or 18)
I Intermed synop hour (rounds to 00/03/06/09/12/15/18/21)
F Frontal hour (for ASUS1 KWBC frontal position bulletin) (rounds to 01/04/07/10/13/16/19/22)
     *** note that some servers do not produce a ASUS1 at 22Z ***

Although Digital Atmosphere automatically figures the current time tokens before you retrieve data, you may want to override them. This is handy if you want to see data for an earlier hour. All you have to do is change the numbers that are displayed in the time token panel.

Solving incorrect time problems. If the time tokens always show a slightly incorrect hour, check your Preferences menu under the General tab and make sure the "Daylight Saving" checkbox is marked in accordance with whether daylight savings time is in effect. If the hours are significantly off, go to your Windows control panel, look under Date/Time, and make sure your computer is set to the correct time zone. Then make sure your system clock is set to a current time which is valid for that time zone. For example, if you live in New York and typically have your computer clock set to Greenwich Mean Time, you will have to set the Windows time zone to "GMT" rather than Eastern Time.

Solving access problems. Digital Atmosphere must have access to a true Winsock Internet connection in order to retrieve data. It may not work with proprietary systems such as AOL and MSN. If for some reason you can't retrieve data through Digital Atmosphere, all is not lost. You can still use your Internet browser to get products, then save them to your Digital Atmosphere data directory, which is most users have had to do up until 1997!

Technical tip. Do you want to swap data product lists with other users? No problem; we've made it easy. Simply give them a copy of your DIGATMOS.SLZ file (it's in your Digital Atmosphere directory). If they replace their DIGATMOS.SLZ file with yours, they'll get your list of data products. We sometimes offer updated DIGATMOS.SLZ files on our web site to give you the latest set of data products.

Note that you can set a password privately by using the DIGATMOS.PWD file (if there isn't one, you may create it).  This will make your scripts secure and prevent passwords from showing up in the status windows. This file must be located in the Digital Atmosphere directory.  Make entries as follows:

token=value

So if you have a login for your blog site, you might have something like this in your DIGATMOS.PWD file:

myusername=joeblow
mypassword=12341234

When Digital Atmosphere encounters %%something%% in your URL, it will replace it with the values you defined in DIGATMOS.PWD.  Thus, in the example above %%myusername%% would get replaced with joeblow, and %%mypassword%% would get replaced with 12341234.