The Weather Forecasting Forum

Discussions about the latest in meteorology, programs, tools, and techniques
It is currently Wed Oct 18, 2017 4:42 pm

All times are UTC




Post new topic Reply to topic  [ 10 posts ] 
Author Message
PostPosted: Thu Dec 30, 2010 10:52 pm 
Offline
Moderator

Joined: Sat Sep 16, 2006 8:08 pm
Posts: 13
Hi everyone,

I read on the scripting forum that there is a change in links for getting MADIS data into Digital Atmosphere. I am quoting Wally Mayo from the scripting forum were they provided links for a certain part of the country:

Quote:
Here's the script for lower MI. If you need a wider area, just change the lat/long parameter in the file.

http://madis-data.noaa.gov/cgi-bin/text ... d=WXforYou

Sorry for the delay. I forgot my last post.

Be sure to COPY LINK Location to get all of the script.

Hmmm, you probably meant ALL of lower MI - try this one

http://madis-data.noaa.gov/cgi-bin/text ... d=WXforYou


I know that these only grab portions of the MADIS data, but I recieved an e-mail today that the MADIS links are changing to these new servers. Digital Atmosphere might need an update to the current links in the program to reflect the new servers.

The new server is madis-data.nws.noaa.gov. So in the post above I guess the links would change to reflect the new server. I just tested them and they worked.

The NEW Link for getting the HUGE file for Digital Atmosphere is as follows:

Code:
http://madis-data.nws.noaa.gov/cgi-bin/textXmlRqt?rdr=http%3A%2F%2Fpublic_madis_public%3A0823p%21nIaB%40madis-data.nws.noaa.gov%2FmadisPublic%2Fcgi-bin%2FmadisXmlPublicDir%3F&time=0&minbck=-60&minfwd=0&recwin=3&dfltrsel=1&state=AK&latll=25&lonll=-130&latur=55&lonur=-60&stanam=&stasel=0&pvdrsel=0&varsel=2&qcsel=1&xml=0


It has been changed to reflect the new URLs and servers at MADIS for the huge file configuration under Digital Atmosphere.

I hope this helps everyone.

Thanks,
Christopher Zenzel


Top
 Profile  
 
PostPosted: Thu Apr 14, 2011 9:07 am 
Offline
Junior Member

Joined: Wed Jun 25, 2008 3:23 am
Posts: 6
Location: Bellevue, NE
I'm running into a little trouble downloading MADIS... I'm hoping someone can help.

In my script, I have DA downloading METAR, CWOP, SYNOP, and MADIS using the following format: download,URL,filename

This works for METAR, CWOP, and SYNOP, but when I get to MADIS, I get a "Cannot create file" error. Here's the script for MADIS:

download,http://madis-data.nws.noaa.gov/cgi-bin/textXmlRqt?rdr=http%3A%2F%2Fpublic_madis_public%3A0823p%21nIaB%40madis-data.nws.noaa.gov%2FmadisPublic%2Fcgi-bin%2FmadisXmlPublicDir%3F&time=0&minbck=-60&minfwd=0&recwin=3&dfltrsel=1&state=AK&latll=25&lonll=-130&latur=55&lonur=-60&stanam=&stasel=0&pvdrsel=0&varsel=2&qcsel=1&xml=0,E:\Weather\WxCharts\svrfloat\madis.txt

If I copy/paste the URL into a web browser, the file opens up. Any ideas on how I can get this data to download/ingest into DA? Thanks!


Top
 Profile  
 
PostPosted: Fri Apr 15, 2011 3:30 am 
Offline
Member

Joined: Sun Nov 30, 2003 5:05 am
Posts: 96
Location: Fort Worth, TX
Quick thoought without checking, but does the "E" drive actually exisit? Is that the correct path in the script to create the madis.txt file? Is there a "space" at the very end? DA DOES NOT like spaces.............


Top
 Profile  
 
PostPosted: Sun Apr 24, 2011 8:18 pm 
Offline
Junior Member

Joined: Wed Jun 25, 2008 3:23 am
Posts: 6
Location: Bellevue, NE
Sorry for the delayed reply...

There are no spaces at the end and the E drive is my local, external hard drive. I have no problems dealing with this drive as ALL my other data runs just fine. Here's the complete download portion of my script...


Code:
download,http://www.atmos.albany.edu/weather/data1/surface/sao/@Y@M@D@H_sao.wmo,E:\Weather\floater\metar.txt
download,http://weather.cod.edu/digatmos/syn/@Y@M@D@I.syn,E:\Weather\floater\synop.txt
download,http://www.madis-fsl.org/cgi-bin/simpleMadisXmlRqt?rdr=http%3A%2F%2Fpublic_madis_public%3A0823p%21nIaB%40www.madis-fsl.org%2FmadisPublic%2Fcgi-bin%2FmadisXmlPublicDir%3F&time=0&minbck=-60&minfwd=0&recwin=3&dfltrsel=1&state=AK&latll=25&lonll=-130&latur=55&lonur=-60&stanam=&stasel=0&pvdrsel=0&varsel=2&qcsel=1&xml=0,E:\Weather\floater\madis.txt
download,http://wxqa.com/activecwd/C20@Y@M@D.txt,E:\Weather\floater\cwop.txt
download,http://www.atmos.albany.edu/weather/data1/surface/boy/@Y@M@D@I_boy.wmo,E:\Weather\floater\buoy.txt


Of course, the madis line breaks on this forum, but in the .dsf file it's one continuous line.


Top
 Profile  
 
PostPosted: Tue May 17, 2011 8:45 pm 
Offline
Member

Joined: Tue Oct 30, 2007 9:08 pm
Posts: 36
Hi folks...

The address for the huge MADAIS file hasn't worked for me since about May 12 or so. Does anybody have a similar experience?

Heatwave


Top
 Profile  
 
PostPosted: Wed May 18, 2011 1:37 am 
Offline
Member

Joined: Sun Nov 30, 2003 5:05 am
Posts: 96
Location: Fort Worth, TX
MADIS has changed the way you log in and the method (now ftp I beleive) to obtain the data effective 12 May. I haven't played with it yet to see what needs to be done.

Greg


Top
 Profile  
 
PostPosted: Sat May 21, 2011 8:06 pm 
Offline
Moderator

Joined: Sat Sep 16, 2006 8:08 pm
Posts: 13
Hey everyone,

I'm back from school work and looked into the issue. I get the new info from MADIS via e-mail and found the new url to be:

Code:
https://madis-data.noaa.gov/cgi-bin/textXmlRqt?rdr=https%3A%2F%2Fpublic_madis_public%3A0823p%21nIaB%40madis-data.noaa.gov%2FmadisPublic1%2Fcgi-bin%2FmadisXmlPublicDir%3F&time=0&minbck=-60&minfwd=0&recwin=3&dfltrsel=1&state=AK&latll=25&lonll=-130&latur=55&lonur=-60&stanam=&stasel=0&pvdrsel=0&varsel=2&qcsel=1&xml=0


Please let me know if it works for you. I currently tested it in the browser and DA seems to be ablr to download it without error, but I might be wrong.

Thanks,
Christopher Zenzel


Top
 Profile  
 
PostPosted: Sat May 21, 2011 8:11 pm 
Offline
Moderator

Joined: Sat Sep 16, 2006 8:08 pm
Posts: 13
czenzel wrote:
Hey everyone,

I'm back from school work and looked into the issue. I get the new info from MADIS via e-mail and found the new url to be:

Code:
https://madis-data.noaa.gov/cgi-bin/textXmlRqt?rdr=https%3A%2F%2Fpublic_madis_public%3A0823p%21nIaB%40madis-data.noaa.gov%2FmadisPublic1%2Fcgi-bin%2FmadisXmlPublicDir%3F&time=0&minbck=-60&minfwd=0&recwin=3&dfltrsel=1&state=AK&latll=25&lonll=-130&latur=55&lonur=-60&stanam=&stasel=0&pvdrsel=0&varsel=2&qcsel=1&xml=0


Please let me know if it works for you. I currently tested it in the browser and DA seems to be ablr to download it without error, but I might be wrong.

Thanks,
Christopher Zenzel



Looks like it might not be loading in Digital Atmosphere yet. Might have to grab the MADIS file with another program and then import it to DA. A program such as wget would work and import with a script. We will need to contact Tim about the SSL madis. The link does work in the browser, just need to do an external get when importing to DA.


Top
 Profile  
 
PostPosted: Sat May 21, 2011 9:52 pm 
Offline
Moderator

Joined: Sat Sep 16, 2006 8:08 pm
Posts: 13
Hi Everyone,

To use MADIS in Digital Atmosphere for now you can do the following:

1.) Download CURL from the offical CURL web site at http://curl.haxx.se/download.html. You will

need to download the appropriate version for your system. The binary that most likely will work

for you is the static Win32 binary with SSL support. Do not just download libcurl but also the

CURL program.

A direct link is provided for the latest version here:

Code:
http://www.gknw.net/mirror/curl/win32/curl-7.21.6-ssl-sspi-zlib-static-bin-w32.zip


2.) Use the following command prompt script to download the latest MADIS file and save it to

your hard disk using curl:

Since CURL doesn't support the redirect that MADIS uses you must decode the MADIS string at the

end (after ?rdr=) and replace the proper user to the new MADIS account. You must also specify

that this is an insecure download.

Do NOT re-encode the URL.

To find a URL encoder/decoder on the Internet you can use the following site:
Code:
http://meyerweb.com/eric/tools/dencoder/


madis.bat:
Code:
set PATH=%PATH%
@echo OFF
echo MADIS Download Script for Digital Atmosphere...
echo Now starting MADIS download with public access account...

curl -L --insecure "https://public_madis_public:0823p!nIaB@madis-

data.nws.noaa.gov/madisPublic1/cgi-bin/madisXmlPublicDir?&time=0&minbck=-

60&minfwd=0&recwin=3&dfltrsel=1&state=AK&latll=25&lonll=-130&latur=55&lonur=-

60&stanam=&stasel=0&pvdrsel=0&varsel=2&qcsel=1&xml=0" > madis.txt

echo Done downloading MADIS information...


3.) Now import the file into Digital Atmosphere. In the same directory create a script:

Make sure to replace PATH_TO_MADIS to the path of your downloaded madis.txt File from

madis.bat!

madis.dsf
Code:
INGEST,PATH_TO_MADIS\madis.txt


To run open Digital Atmopshere and go to the Scripting tab and click "Run" and run the

"madis.dsf" file after proper modifications are made to the script.

This will take a while with the HUGE MADIS file!

4.) Now try a sample plot to make sure data has imported into the program from MADIS.

Now you are able to use MADIS data. Feel free to modify this in any way you see for your

application. The only requirement is that you need is CURL for Windows with the SSL libraries.

Enjoy!
Christopher Zenzel


Top
 Profile  
 
PostPosted: Wed Jun 15, 2011 11:47 am 
Offline
Member

Joined: Wed Jun 15, 2011 11:44 am
Posts: 50
I found good us in
Workspace Macro
at
tethyssolutions.com

It is extremely simple in operation, and makes getting MADIS very easy.
At least until the real fix in DA is done.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 10 posts ] 

All times are UTC


Who is online

Users browsing this forum: Yahoo [Bot] and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group