NCDC Needs An Overhaul

[HAltschule]

Wow. For those of you who use the National Climatic Data Center regularly to retrieve data, you likely have noticed that the entire computer system crashed last Wednesday and still, as of this 2pm Tuesday writing, remains down. That means that anyone in the U.S. that is trying to access the system to retrieve past weather data or doppler radar archives is out of luck.

It amazes me that the Director and management of the NCDC allow these kinds of computer issues to occur so often. And it does happen often. Maybe it's time for some changes at the NCDC? Afterall, these kinds of problems affect those of us in the weather consulting business...or those of you who are trying to do some research. I, for one, don't like to explain to my clients why their work is being delayed for so long.

The quality of the data has decreased as well. I have come across numerous errors in the "Edited" data that the NCDC provides...sometimes 2 different high temperature readings from the station for the same day...but on 2 different documents. WTF?

Anyway, I'm curious if anyone else has some thoughts on the NCDC and their customer service over the past year or two.

 

[rdale]

Are you sure it's management and not budget? If they don't have the money to support the archiving, I'd think that is a Congressional problem and not leadership...

 

[HAltschule]

Not 100% sure. But the decision last year not to archive the "comments" sections of the METAR's for dates earlier than 2005 is ridiculous. That would be a management decision and a reason to question the leadership. Since they realized how bad a decision that was, they are having their people include them from 2005 to present. But they have no intention of going back and adding back in prior to 1/1/2005. So, we lose some crucial data. There are many other issues I have also...probably because I use their services daily. Many of the folks I deal with are great and very helpful, some are not. But the bottom line is, my work is at a standstill directly because of the NCDC.

Oh well, venting complete for now.

 

[Melissa Moon]

There are firm plans to reorganize the divisions here, but before any of that can officially take place, it must go through NOAA...that doesn't happen overnight, but there is an "overhaul" in the works.

Also, funding hasn't been as good the past 2 or so years as it's been in the past. I don't know who made the METAR decision, but chances are the reason they're not going back to do those comments is because of money...there's so much to do with the data (we have TONS that have not even been scanned or keyed yet), and the personnel NCDC can afford to have can only do much with our time, so those in charge of assigning tasks have to prioritize what is most important at the present time. Right now, METAR comments may take a lower priority as something like getting more ships data keyed. it doesn't mean they will never get done...it just means that with the current budget they're having to pick and choose what to do with the resources that we have since it is limited.

And as someone who's had to get their hands dirty actually looking at a lot of the forms myself, sometimes you have instances where you have two forms where the observer wrote identical dates on the forms and there is no way of telling which one is correct (though usually it's blatantly obvious). When that happens, we're not supposed to make changes unless we know for certain. I don't work with the "edited" data that you talk about (all the stuff I did was on WSSRD), so I don't know their policies, but I do know that sometimes that does happen and there's nothing anyone can do about it.

As for the computer issue, I've been getting Emails about it all week...they had problems with some of the hardware and have been working nonstop to fix it. I'm not sure if you know much about the supercomputer at NCDC, but it's one of the largest in the world and because so much data is constantly coming in, they're constantly having to expand stuff. The thing is massive...so because it's so big and we're dealing with so much data, that unfortunately means that there are more opportunities for things to go wrong. And at the same time we've got tons of people trying to hack us, which doesn't help.

Sadly I cannot tell you when they are going to have stuff back and running again with the computer...I don't work in IT...but from the Emails I've been getting it sounds to me like the problem was pretty serious (but not anyone's fault). So, it's not like people are sitting around drinking their java while everything breaks...those guys upstairs constantly have their hands full maintaining a system as large as NCDC.

This probably doesn't make you feel any better :-\...it stinks being the middleman between the data and your customer...but I wanted to leave a few remarks since I spend a third of my day each weekday here.

 

[Melissa Moon]

I should note i am not "officially" speaking on behalf of NCDC to anyone on here...everything above is my own observation and thoughts

 

[Zac Flamig]

I, for one, don't like to explain to my clients why their work is being delayed for so long.

Sounds to me like you need to invest in making your own database of archived information then.

Afterall, if you take a quick glance at NCDC's webpage you can clearly see that their mission statement is to provide access to this information. It doesn't specify a timeline for how fast access to it shall be given.

 

[HAltschule]

Hi Melissa,

Nice post, good information to know. I do understand that the budget has come into play and that there are priorities. I understand from some conversations today that the NCDC has computer / IT people from California on the phone trying to fix things. Anyway, I was told tentatively that it will be back up on Wednesday or Thursday "hopefully". Those were the words. That would be a full week, I think.

Having been dealing with the NCDC folks for the past 9 years I have learned who I like to work with (and who knows what they're doing). The ones who know what they're doing get me exactly what I need in no time. I'm sure you are in that "good" category even though we haven't spoken yet.

Anyway, I do appreciate your comments and insight. I am in the process of looking into a private weather NOAAPORT feed to import all of the data and archive it myself. Hopefully I won't have so many computer problems as NCDC. After all, I am still unable to work on my 8 cases right now and I have 4 trials to prepare for over the next 7 days. (I sure hope the system is back up SOON).

Good luck with the customer calls...

 

[Donald Giuliano]

Howard,

I might be able to help you out here, as it's possible I might have access to at least some of the data you are seeking that is normally hosted on NCDC's website. If you want you can send me a PM with the dates and data types you need and I'll let you know what I have.

 

[Melissa Moon]

Hi Melissa,

Nice post, good information to know. I do understand that the budget has come into play and that there are priorities. I understand from some conversations today that the NCDC has computer / IT people from California on the phone trying to fix things. Anyway, I was told tentatively that it will be back up on Wednesday or Thursday "hopefully". Those were the words. That would be a full week, I think.

Having been dealing with the NCDC folks for the past 9 years I have learned who I like to work with (and who knows what they're doing). The ones who know what they're doing get me exactly what I need in no time. I'm sure you are in that "good" category even though we haven't spoken yet.

Anyway, I do appreciate your comments and insight. I am in the process of looking into a private weather NOAAPORT feed to import all of the data and archive it myself. Hopefully I won't have so many computer problems as NCDC. After all, I am still unable to work on my 8 cases right now and I have 4 trials to prepare for over the next 7 days. (I sure hope the system is back up SOON).

Good luck with the customer calls...

Haha...you probably won't ever talk to me when you call needing something. I don't work in customer service. I work with the Forts data (pre-1892). Do you have a WSSRD account? Anyone can get one from what I understand...