Setting up a wxserver: GEMPAK, LDM, McIDAS, etc.???

[Darren Addy]

Hi Guys,
I've got a free Linux Server lying around here at the university and thought it might be cool to set up a wxserver for our Geography department and their student organization. (What can I say? I'm an altruistic kinda guy! :roll: ) I'm considering installing LDM, McIDAS & GEMPAK and maybe a GIS mapserver.

I believe that as an educational institution, although some of the data will need to be restricted campus, we can get most data streams free (is that correct?)

The one thing I will need to do is upgrade the hard drive. Any idea what capacity I should get? I see mentions of LDM cache sizes of 2 GB and want to make sure I get enough capacity for everything.

If anybody has been down this road and would like to share their wisdom, I'd appreciate it. (Other programs to consider, best data sources -particularly for the stormchaser's game- ect.)

Thanks in advance!

Darren Addy
Kearney, NE

 

[Ben Cotton]

I believe that as an educational institution, although some of the data will need to be restricted campus, we can get most data streams free (is that correct?)

Correct. If you're already affiliated with Unidata then you can e-mail [email protected] to get a feed assigned. On the other hand, since I'm a nice guy myself, when you get your machine set up, send me the name and I'll feed you from Purdue's machine. I can provide you just about everything (ironically, you'll probably have to contact someone else for Level II data, someone else maintains those machines and I think he's at the limit of the number of downstream sites he wants.). Lightning data is tightly restricted -- only SUNY Albany (is it just U of Albany now?) can distribute that. You can contact David Knight ([email protected], I think) to set that up.

The one thing I will need to do is upgrade the hard drive. Any idea what capacity I should get?
Our old machine had an 80GB drive and that was good for our purposes. If you want to keep data around a long time or go for the bigger data sets, you'll want more. We've got about 400GB now and that's plenty.

Let me know if there's anything else I can do to help!


Ben

 

[Darren Addy]

Ben,
You sure took your sweet time answering. I was tapping my toe for a solid TEN MINUTES there! :lol:

Thanks so much for your kind offer. I may just take you up on that. I have a my.unidata log-in, so does that mean I'm affiliated?

Does my list of programs sound right to you? Any others you might recommend? Do you also run a GIS mapserver?

(I will probably have to wait for the larger HD to come in before I can start this in earnest, but hopefully that won't be longer than a week.)

Thanks again for the helpful (and fast) response. I love this forum!

Darren Addy
Kearney, NE

 

[Ben Cotton]

Thanks so much for your kind offer. I may just take you up on that. I have a my.unidata log-in, so does that mean I'm affiliated?

If your institution (I'm not sure what it is ) is listed here http://my.unidata.ucar.edu/content/communi...iversities.html then you're affiliated. If not, I'm sure it wouldn't take much for you to get in. Unidata isn't quite a secret society.

Does my list of programs sound right to you? Any others you might recommend?

Except for a few decoders, McIDAS isn't too vital...I can't think of anyone that has used it in the past 10 years, but I could be wrong. Just with LDM and Gempak, you'd be in good shape. Myself, I'm partial to WXP but that might be a bit costly for you. Purdue gets it for free because we developed it, then sold it (and Dan Vietor) to Unisys. I don't know if Unisys offers special deals to universities or not.

Do you also run a GIS mapserver?

Nope.


Ben

 

[Robert Dewey]

Interesting...

I am officially a "Unidata Sponsored Individual"... GEMPAK and LDM are free to everyone, McIDAS is not, you need special permission from the University of Wisconsin - I was lucky enough to obtain it, but it's pretty much just a fancy satellite viewer, and you most likely won't make any use of it...

I am currently taking in nearly 1.5GB/hr, which gives me most of the CONDUIT stuff, the NOAAport stream, numerous Level II NEXRAD sites, ACARS data, etc.. You're gonna need bandwidth. My latency is relatively high at 1.5GB/hr, running about 1000 seconds for CONDUIT... But it's still faster than most website GIF images. I am running a ~5-7Mbit/sec connection... Heck, I'm surprised my ISP hasn't contacted me about the bandwidth usage. You're site is probably running a T3, but with all of the other systems (and/or dorms) connected, bandwidth could be an issue.

The main institution for getting L2 data would be TAMU (Texas A&M)... Gilbert (weather2.admin.niu.edu) is also friendly when it comes to obtaining data. You should certainly take Ben up on his offer too. Most sites just want to know what you plan on doing with the data (non commercial/non operational/educational/etc.).

But... Since you're an educational institution, you should contact Unidata and ask them to set you up. They will put you in contact with all of the sites/data you are requesting. You should also contact you're university/college system admin about getting things on the network as well. NLDN data should be available to you, as well as other proprietary data sources.

All in all, my server didn't cost too much to build. It's 1.5GHz AMD Duron with 2GB RAM and an 80GB standard 7200RPM hard drive. It has never had a crash (other than me trying to upgrade something, and breaking it). It runs 2 NIC devices, one for inbound traffic, and the other for outbound sharing of data and webserver stuff.

 

[Ben Cotton]

I am currently taking in nearly 1.5GB/hr, which gives me most of the CONDUIT stuff, the NOAAport stream, numerous Level II NEXRAD sites, ACARS data, etc.. You're gonna need bandwidth.

Darren, to start with, I'd leave off CONDUIT and the Level II feed. The HDS model feed and the Level III data coming in on NNEXRD should be plenty. If you have the available bandwidth, drive space, and system resources, give it a whirl, but starting out with a smaller data set (HDS instead of CONDUIT and NNEXRD instead of CRAFT) might make life a little simpler. That'll also help keep latencies down until you can find out what your machine is really capable of handling.


Ben

 

[Tyler Allison]

You may also want to get in on Unidata's grant program that just started up again a few weeks ago.

http://my.unidata.ucar.edu/content/communi...rd/RFP2005.html

They'll pay for your equipment and what not if your grant is approved. It'd be worth submitting just in case.

rdale (one of the members here) has a commercial Gempak 'setup' offering so it may be worth talking to him about helping with setup, etc. You may want to just get Gempak installed and grab the data "as needed"...and then worry about an LDM feed. rdale's systems come with FTP/web scripts for everything you need

 

[Darren Addy]

You may also want to get in on Unidata's grant program that just started up again a few weeks ago.

Well, shoot. Looks like i'm 10 days past their submission deadline right now. Might keep that in mind if they repeat it next year, though!
It was an post of rdales that first made me aware of GEMPAK. I was hoping that I would have enough server experience to set it up on my own (and maybe learn something). But it is nice to know that I'm working with a net. Thanks, Tyler!

Thanks also to Robert Dewey and Ben Cotton for the data source and bandwidth-consideration info.

Question: In this case, what do we mean when we talk about latency issues? Is it the difference between real time and processing/rendering time? Or is there more to it than that?

Darren Addy
Kearney, NE

 

[Robert Dewey]

Question: In this case, what do we mean when we talk about latency issues? Is it the difference between real time and processing/rendering time? Or is there more to it than that?

Darren Addy
Kearney, NE

Latency is the time the product was injected into the feed (either by Unidata or NCEP/NWS) and the time you receive it.

 

[Tyler Allison]

Also don't get all caught up with the actual time listed on the file. The NWS does not use the atomic clock, so I've heard, so it may look like you got the file 4 or 5 minutes after they released it but it may really only be 20 seconds because the radar stations clock is all screwy.

This confuses the heck out of folks using the online radar products. They complain to the vendor that the data feed is slow based on the NWS timestamp.

I've heard some of the radar products are timestamped when the radar starts it's rotation...which would mean the data looks like it's 60 seconds old by the time it hits the wire...then 5 more seconds to hit the injection locations (Unidata or NWS) and then another 2 seconds to get to a Teir-2 location or the NWS servers and then another second or to for you to get it assuming you can time sync that closely with your upstream service. And that's assuming the Internet is being cooperative that night

If you can afford a Tier-1 provider they claim less than 10 second latency..which is 10 seconds from the time the data comes off the radar computer to your hot little hands. Of course that comes with a price

 

[Ben Cotton]

If you can afford a Tier-1 provider they claim less than 10 second latency..which is 10 seconds from the time the data comes off the radar computer to your hot little hands. Of course that comes with a price

The price is free. We can only charge private sector customers, and then "at cost." We're required to pass it along free to other Unidata members.

 

[Tyler Allison]

If you can afford a Tier-1 provider they claim less than 10 second latency..which is 10 seconds from the time the data comes off the radar computer to your hot little hands. Of course that comes with a price

The price is free. We can only charge private sector customers, and then "at cost." We're required to pass it along free to other Unidata members.

True..I meant non-Unidata folks. You lucky punks get all the good toys

For us private sector folks..."at cost" is still above what most storm chasers would be willing to pay. Im skirting the edges of forum rules here with my next sentance but it does make a point...even on my low budget L2 system, $30 a month aint 'free'

 

[Ben Cotton]

True..I meant non-Unidata folks. You lucky punks get all the good toys

Why do you think I don't want to graduate? Just kidding, get me the heck out of here.

For us private sector folks..."at cost" is still above what most storm chasers would be willing to pay. Im skirting the edges of forum rules here with my next sentance but it does make a point...even on my low budget L2 system, $30 a month aint 'free'

Aye, ain't that the truth. Sometimes you have to wonder how much "cost" my counterparts in the Research Computing Department (or whatever they're calling themselves these days) are really incurring. Ah well, universities are nothing but state-funded corporations these days anyway. </threadjack>


Ben