Page 1 of 1

New MADIS Links - Digital Atmosphere has OLD Links

Posted: Thu Dec 30, 2010 10:52 pm
by czenzel
Hi everyone,

I read on the scripting forum that there is a change in links for getting MADIS data into Digital Atmosphere. I am quoting Wally Mayo from the scripting forum were they provided links for a certain part of the country:
Here's the script for lower MI. If you need a wider area, just change the lat/long parameter in the file.

http://madis-data.noaa.gov/cgi-bin/text ... d=WXforYou

Sorry for the delay. I forgot my last post.

Be sure to COPY LINK Location to get all of the script.

Hmmm, you probably meant ALL of lower MI - try this one

http://madis-data.noaa.gov/cgi-bin/text ... d=WXforYou
I know that these only grab portions of the MADIS data, but I recieved an e-mail today that the MADIS links are changing to these new servers. Digital Atmosphere might need an update to the current links in the program to reflect the new servers.

The new server is madis-data.nws.noaa.gov. So in the post above I guess the links would change to reflect the new server. I just tested them and they worked.

The NEW Link for getting the HUGE file for Digital Atmosphere is as follows:

Code: Select all

http://madis-data.nws.noaa.gov/cgi-bin/textXmlRqt?rdr=http%3A%2F%2Fpublic_madis_public%3A0823p%21nIaB%40madis-data.nws.noaa.gov%2FmadisPublic%2Fcgi-bin%2FmadisXmlPublicDir%3F&time=0&minbck=-60&minfwd=0&recwin=3&dfltrsel=1&state=AK&latll=25&lonll=-130&latur=55&lonur=-60&stanam=&stasel=0&pvdrsel=0&varsel=2&qcsel=1&xml=0
It has been changed to reflect the new URLs and servers at MADIS for the huge file configuration under Digital Atmosphere.

I hope this helps everyone.

Thanks,
Christopher Zenzel

Re: New MADIS Links - Digital Atmosphere has OLD Links

Posted: Thu Apr 14, 2011 9:07 am
by senorpepr
I'm running into a little trouble downloading MADIS... I'm hoping someone can help.

In my script, I have DA downloading METAR, CWOP, SYNOP, and MADIS using the following format: download,URL,filename

This works for METAR, CWOP, and SYNOP, but when I get to MADIS, I get a "Cannot create file" error. Here's the script for MADIS:

download,http://madis-data.nws.noaa.gov/cgi-bin/ ... \madis.txt

If I copy/paste the URL into a web browser, the file opens up. Any ideas on how I can get this data to download/ingest into DA? Thanks!

Re: New MADIS Links - Digital Atmosphere has OLD Links

Posted: Fri Apr 15, 2011 3:30 am
by Greg Higgins
Quick thoought without checking, but does the "E" drive actually exisit? Is that the correct path in the script to create the madis.txt file? Is there a "space" at the very end? DA DOES NOT like spaces.............

Re: New MADIS Links - Digital Atmosphere has OLD Links

Posted: Sun Apr 24, 2011 8:18 pm
by senorpepr
Sorry for the delayed reply...

There are no spaces at the end and the E drive is my local, external hard drive. I have no problems dealing with this drive as ALL my other data runs just fine. Here's the complete download portion of my script...

Code: Select all

download,http://www.atmos.albany.edu/weather/data1/surface/sao/@Y@M@D@H_sao.wmo,E:\Weather\floater\metar.txt
download,http://weather.cod.edu/digatmos/syn/@Y@M@D@I.syn,E:\Weather\floater\synop.txt
download,http://www.madis-fsl.org/cgi-bin/simpleMadisXmlRqt?rdr=http%3A%2F%2Fpublic_madis_public%3A0823p%21nIaB%40www.madis-fsl.org%2FmadisPublic%2Fcgi-bin%2FmadisXmlPublicDir%3F&time=0&minbck=-60&minfwd=0&recwin=3&dfltrsel=1&state=AK&latll=25&lonll=-130&latur=55&lonur=-60&stanam=&stasel=0&pvdrsel=0&varsel=2&qcsel=1&xml=0,E:\Weather\floater\madis.txt
download,http://wxqa.com/activecwd/C20@Y@M@D.txt,E:\Weather\floater\cwop.txt
download,http://www.atmos.albany.edu/weather/data1/surface/boy/@Y@M@D@I_boy.wmo,E:\Weather\floater\buoy.txt
Of course, the madis line breaks on this forum, but in the .dsf file it's one continuous line.

Re: New MADIS Links - Digital Atmosphere has OLD Links

Posted: Tue May 17, 2011 8:45 pm
by lrdheat
Hi folks...

The address for the huge MADAIS file hasn't worked for me since about May 12 or so. Does anybody have a similar experience?

Heatwave

Re: New MADIS Links - Digital Atmosphere has OLD Links

Posted: Wed May 18, 2011 1:37 am
by Greg Higgins
MADIS has changed the way you log in and the method (now ftp I beleive) to obtain the data effective 12 May. I haven't played with it yet to see what needs to be done.

Greg

Re: New MADIS Links - Digital Atmosphere has OLD Links

Posted: Sat May 21, 2011 8:06 pm
by czenzel
Hey everyone,

I'm back from school work and looked into the issue. I get the new info from MADIS via e-mail and found the new url to be:

Code: Select all

https://madis-data.noaa.gov/cgi-bin/textXmlRqt?rdr=https%3A%2F%2Fpublic_madis_public%3A0823p%21nIaB%40madis-data.noaa.gov%2FmadisPublic1%2Fcgi-bin%2FmadisXmlPublicDir%3F&time=0&minbck=-60&minfwd=0&recwin=3&dfltrsel=1&state=AK&latll=25&lonll=-130&latur=55&lonur=-60&stanam=&stasel=0&pvdrsel=0&varsel=2&qcsel=1&xml=0
Please let me know if it works for you. I currently tested it in the browser and DA seems to be ablr to download it without error, but I might be wrong.

Thanks,
Christopher Zenzel

Re: New MADIS Links - Digital Atmosphere has OLD Links

Posted: Sat May 21, 2011 8:11 pm
by czenzel
czenzel wrote:Hey everyone,

I'm back from school work and looked into the issue. I get the new info from MADIS via e-mail and found the new url to be:

Code: Select all

https://madis-data.noaa.gov/cgi-bin/textXmlRqt?rdr=https%3A%2F%2Fpublic_madis_public%3A0823p%21nIaB%40madis-data.noaa.gov%2FmadisPublic1%2Fcgi-bin%2FmadisXmlPublicDir%3F&time=0&minbck=-60&minfwd=0&recwin=3&dfltrsel=1&state=AK&latll=25&lonll=-130&latur=55&lonur=-60&stanam=&stasel=0&pvdrsel=0&varsel=2&qcsel=1&xml=0
Please let me know if it works for you. I currently tested it in the browser and DA seems to be ablr to download it without error, but I might be wrong.

Thanks,
Christopher Zenzel

Looks like it might not be loading in Digital Atmosphere yet. Might have to grab the MADIS file with another program and then import it to DA. A program such as wget would work and import with a script. We will need to contact Tim about the SSL madis. The link does work in the browser, just need to do an external get when importing to DA.

MADIS Link Update Tutorial (SSL)

Posted: Sat May 21, 2011 9:52 pm
by czenzel
Hi Everyone,

To use MADIS in Digital Atmosphere for now you can do the following:

1.) Download CURL from the offical CURL web site at http://curl.haxx.se/download.html. You will

need to download the appropriate version for your system. The binary that most likely will work

for you is the static Win32 binary with SSL support. Do not just download libcurl but also the

CURL program.

A direct link is provided for the latest version here:

Code: Select all

http://www.gknw.net/mirror/curl/win32/curl-7.21.6-ssl-sspi-zlib-static-bin-w32.zip
2.) Use the following command prompt script to download the latest MADIS file and save it to

your hard disk using curl:

Since CURL doesn't support the redirect that MADIS uses you must decode the MADIS string at the

end (after ?rdr=) and replace the proper user to the new MADIS account. You must also specify

that this is an insecure download.

Do NOT re-encode the URL.

To find a URL encoder/decoder on the Internet you can use the following site:

Code: Select all

http://meyerweb.com/eric/tools/dencoder/
madis.bat:

Code: Select all

set PATH=%PATH%
@echo OFF
echo MADIS Download Script for Digital Atmosphere...
echo Now starting MADIS download with public access account...

curl -L --insecure "https://public_madis_public:0823p!nIaB@madis-

data.nws.noaa.gov/madisPublic1/cgi-bin/madisXmlPublicDir?&time=0&minbck=-

60&minfwd=0&recwin=3&dfltrsel=1&state=AK&latll=25&lonll=-130&latur=55&lonur=-

60&stanam=&stasel=0&pvdrsel=0&varsel=2&qcsel=1&xml=0" > madis.txt

echo Done downloading MADIS information...
3.) Now import the file into Digital Atmosphere. In the same directory create a script:

Make sure to replace PATH_TO_MADIS to the path of your downloaded madis.txt File from

madis.bat!

madis.dsf

Code: Select all

INGEST,PATH_TO_MADIS\madis.txt
To run open Digital Atmopshere and go to the Scripting tab and click "Run" and run the

"madis.dsf" file after proper modifications are made to the script.

This will take a while with the HUGE MADIS file!

4.) Now try a sample plot to make sure data has imported into the program from MADIS.

Now you are able to use MADIS data. Feel free to modify this in any way you see for your

application. The only requirement is that you need is CURL for Windows with the SSL libraries.

Enjoy!
Christopher Zenzel

Re: New MADIS Links - Digital Atmosphere has OLD Links

Posted: Wed Jun 15, 2011 11:47 am
by Wally
I found good us in
Workspace Macro
at
tethyssolutions.com

It is extremely simple in operation, and makes getting MADIS very easy.
At least until the real fix in DA is done.