Criticism of forecasts

[Tim Vasquez]

Not wanting to bash anyones forecast at NHC but it the eye tilted that much where the long range picks it up over the center of Cuba and the surface center is not even on land?

I saw Doug's post in the hurricane forum and it got me wondering why we are so hesitant to pick apart forecasts. I've certainly noticed that meteorologists in general have gotten much more touchy and somewhat less receptive to constructive criticism than in decades past.

Presumably one reason is that we're all more interconnected, thanks to the Internet, and public criticism has a way of spreading far and wide and sticking for a long time. So everyone's concerned about their public image. Then there's privatization, which brings a conflict of interest. The attorneys at AccuWeather would certainly take a dim view of anyone criticizing their forecasts.

It also appears there's more career advancement going on than in the 1960s/70s, so instead of an "old hat" working in a WFO you now have those same people advancing into management or sidestepping to build experience. Others quit to take care of family, something that would have been rare in the 1960s when dual earner households were not common. So this increases the chance that forecast criticism will land on someone with limited experience and more technical insecurity.

Then there's formal reviews of weather events. These have strong overlap with constructive criticism. I've noticed these have declined substantially since the 1990s and there's often very little mention of official forecast performance. The ones I've seen have generally been biased to highlight forecast successes or have been neutral. It's clear there's a stigma in publishing papers that go against the grain.

Here on Target Area, I've noticed we don't criticize and quarterback each others' forecasts. Why? Because once we start picking apart another person's forecast, it's seen as provocative. It would never happen gracefully unless someone spoke up and said "hey guys, look at my forecast... what did I do wrong?" Once the situation unfolds we seem to talk about the chase and move on.

There seems to be a fundamental disconnect here. Learning from the mistakes of ourselves and others is a huge part of developing the framework of our critical thinking. Our first ten years of life is shaped that way. Since humans have advanced from the ancient days through a long pattern of survival and sustenance where everyday mistakes can bring dire consequences, I'd argue that our brains are specially coded to learn much more efficiently that way.

What to do about it, though, I have no idea. It's a systemic problem; it doesn't even have a solution on Stormtrack as picking apart the existing forecasts here is a recipe for confrontation and flame wars. But there does seem to be a mandate for talking about bad forecasts, especially that of the official agencies, providing it can be handled sensibly and tactfully. We do come here to learn and improve.

Tim

 

[Adam Lucio]

The problem with online critiquing IMO is everything we read we see in plain text. Also I think some people fear a critique will eb viewed as an attack thus subjecting them to penalties and violations.

When critiquing in person you can get a sense of a persons mood, attitude, tone. You can see and hear that theyre engaging in constructive and sometimes helpfull criticism.

Online, its easy to mis-read someones statement and take it as a personal attack against you.

Like if I were to type "If you would check the NAM instead of the GFS, you would see a difference in the storms path" Someone might think Im saying that with a bitter "Im better than you...you moron who doesnt know how to forecast" type attitude.

I guess thats why these forums have smiley faces.

"you're an @$$h@le"....and "you're an @$$H@LE " have two different meanings.

These days allot of peopel are concerned about their image, I can see if its for career purposes, but IMO thats all that should matter, I dont care what people think of me personally. The more you care about your image the more likey you are to get defensive....which can also make you seem like a jerk.

Ah what a vicious cycle....thats my .02 on the subject.

 

[rdale]

Online, its easy to mis-read someones statement and take it as a personal attack against you.

Like if I were to type "If you would check the NAM instead of the GFS, you would see a difference in the storms path" Someone might think Im saying that with a bitter "Im better than you...you moron who doesnt know how to forecast" type attitude.

Can I get an "AMEN" for that! Some people are more direct (i.e. just to the point, like in your first line) vs "Hey, it's a nice day out today, hope you are feeling fine, and oh, by the way, I think the NAM initialized better so have an instinct that it might have the path better." If people are on this forum and talking and contributing, odds are good that the meaning is intended well no matter how it's worded.

Yet SO many people (not just here, but online in general) AUTOMATICALLY assume that the "criticizer" intends the moron statement.

 

[David Wolfson]

My 2-cents as a non-pro who likes to backseat drive.... A lot of short-term forecasting precision seems to depend on precision in determining current conditions, i.e. "Where/what/how much is X?" Give a pro all the data and you get a good forecast.

Since fey Fay inspires the question, I think the reluctance to criticize forecasts comes from an appropriate (I think) understanding that they're limited more than anything else by imperfect horizontal and vertical data. Even right now, where's Fay's center and motion vector? ... Just southeast of Key West and moving maybe 340 at 15-20; but I peeked at the Level II. A few hours ago all you could really tell is that the circulation had come out into the Florida Straits.

With respect to ST, the wide differences among members in experience, knowledge and data access unavoidably add a lot of ad hominem to direct criticism. On the other hand IMO reading the uninhibited forecasts of others is great criticism without being confrontational. FWIW.

 

[Bob Schafer]

A couple things come to my mind on this subject.

Over the last few years, ST has grown exponentially in membership, has it not? With that success comes "fringe" members. There's one in every crowd. Except, now, there's 25 in every crowd.

I think an awful lot of members have ceased/decreased posting so as to avoid criticism and flame wars. I know I've been posting much less frequently the last few years. And, it's not so much that I can't take criticism as that I just don't feel like dealing with that stuff. It's, like, "Ah screw it, I'm just not gonna post anything". And when we DO post, we're now hyper careful to avoid saying anything with the slightest possibility for irritating anyone.

Of course, there are members who are exceptions to this, and you know who you are, LOL.

Every spring I get sick of reading all the forecasts that conclude with "but the models will change..." over and over, like a legal disclaimer that must be included with every post, lest someone reply with "It's early. The models are going to change."

It's analagous to this whole country, really, and the way the lawyers now run it. I recently asked someone who works for the road department around here why it is we were able to BUILD the national highway system, and now we supposedly can't afford to even MAINTAIN those roads with the gas taxes being paid by ten times as many vehicles. He said it's because nowadays every improvement requires a half dozen environmental impact statements, and that THEY cost MORE than any actual construction.

Whoops, did I stray OT? Point is, everything is so insanely PC now.

 

[Bob Hall]

Great subject; I’m not sure anyone other than Tim could have broached it so succinctly as to encourage frank discussion into a potentially loaded subject. Among developmental and learning experts there is a vast and overwhelming consensus that humans of all ages learn far more from their failures than their successes.

If you look at the success of the NHC over the past 25-30 years, there is a general improvement in trajectory projection of one to two percent a year on average. Stated another way in 25 years they have gotten twice as good at what they do. Granted hurricanes are large scale events and the forecasting has been greatly augmented by satellite, airborne recognizance, buoy networks, computer models, radar and other advances.

I postulate that one of the key reasons for the progression is the limited number of storms each year. With between 15 and 20 case studies a year, and a vast data set it is possible to conduct detailed studies and lessons learned exercises to advance the state-of-the-art.

Not to sound too much like a Bill Paxton character, but what frustrates me is the lack of progress we have made in identifying potentially tornadic supercells from those that just scare people. I attended my first spotter training in 1997 and we learned about rear flank downdraft and this clear slot thing. I have not seen the state-of-the-art advance in synoptic scale tornado prediction in the past 11 years.

 

[MClarkson]

If someone wants to question a forecast before or during an event, not a problem. Forecasters argue. That is what we do. Sometimes its even fun. Its the monday morning quarterbacking by someone that had no opinion beforehand that annoys me.

 

[Gene Moore]

Models change or they may not, in the end everyone that actually gets out there and chases must pick a spot. Regardless of the model runs, others endless opinions and mesoscale discussions we can all miss a great event by minutes, miles, or that cell just to the north. In the early years of chasing that's why we got together to share images, to see what was under that other cloud, the one we missed. That concept still lived as we first posted on Stormtrack until Gilbert first put together the Storm Chasers Homepage.

As for forecasting to a public forum, some find it great fun yet the consequences may be 600 other chasers driving to that exact spot. Seldom is a forecast remembered, or the name under it as ever changing technology alters our thinking even moments before we finally commit to an updraft. With all the gadgets and electronic junk and endless calls to others giving guidance we still miss storms and tornadoes. Imagine if you will that we get our wish for perfect models, perfect forecasts. It would be like buying a ticket to a football game, (scalpers would get the best wedge tickets early), we wait in line under the flank, find a parking spot and wait for the show to start.....our tornado will be an elephant trunk at 1645 to 1705 when the trailer park goes airborne. I'm so glad it's not that way yet....someone missed Manchester, missed Attica, missed El Reno, missed Moore, missed Mulvane (I did with a hole in my radiator).

It's a good thing and I hope we don't lose all the mystery in my lifetime.

 

[Paul Knightley]

I'm glad to inform you that if you're a private weather company and you happen to get a forecast wrong, the client seems to take some delight in ringing up and letting you know!

 

[Shane Adams]

Models change or they may not, in the end everyone that actually gets out there and chases must pick a spot. Regardless of the model runs, others endless opinions and mesoscale discussions we can all miss a great event by minutes, miles, or that cell just to the north. In the early years of chasing that's why we got together to share images, to see what was under that other cloud, the one we missed. That concept still lived as we first posted on Stormtrack until Gilbert first put together the Storm Chasers Homepage.

As for forecasting to a public forum, some find it great fun yet the consequences may be 600 other chasers driving to that exact spot. Seldom is a forecast remembered, or the name under it as ever changing technology alters our thinking even moments before we finally commit to an updraft. With all the gadgets and electronic junk and endless calls to others giving guidance we still miss storms and tornadoes. Imagine if you will that we get our wish for perfect models, perfect forecasts. It would be like buying a ticket to a football game, (scalpers would get the best wedge tickets early), we wait in line under the flank, find a parking spot and wait for the show to start.....our tornado will be an elephant trunk at 1645 to 1705 when the trailer park goes airborne. I'm so glad it's not that way yet....someone missed Manchester, missed Attica, missed El Reno, missed Moore, missed Mulvane (I did with a hole in my radiator).

It's a good thing and I hope we don't lose all the mystery in my lifetime.

I don't think we'll ever lose that mystery completely, as there will always be those of us who view chasing as more of a personal adventure/challenge than an investment. Those who choose not to look at every morsel of data and instead, purposely throw caution to the wind because they don't want a guarantee. Like Gene described, it would be so easy if every event was perfectly predictable. For me, it wouldn't be the crowds, but the lack of failure. The stacked odds of successful tornado chasing is half the draw; knowing I'm likely to fail makes those tornadic triumphs so much more satisfying.

As for forecasting, when we can pour a glass of water on the ground and predict (with 100% accuracy) every square inch of ground that will be saturated beforehand, then we can expect perfect performances. After all, it's all just fluid.

 

[rdale]

I attended my first spotter training in 1997 and we learned about rear flank downdraft and this clear slot thing. I have not seen the state-of-the-art advance in synoptic scale tornado prediction in the past 11 years.

It's all about the $$$... Markowski and gang came up with the warm(er) downdraft thing. No way to measure though. CASA has blanketed radars. No money to spread it though. Dual-pol lets us watch inside the storm. Two more years (at least) until we have the money, and even then it's doubtful the data will get outside of the NWS offices because -- no money for bandwidth.

The state-of-the-art advances are still being made. It's just not so easy to translate that to the field without money.

 

[Glen Romine]

Not to sound too much like a Bill Paxton character, but what frustrates me is the lack of progress we have made in identifying potentially tornadic supercells from those that just scare people. I attended my first spotter training in 1997 and we learned about rear flank downdraft and this clear slot thing. I have not seen the state-of-the-art advance in synoptic scale tornado prediction in the past 11 years.

To add to what Rob stated, it is worth recognizing that the emphasis within the NWS remains improving detection (POD - probability of detection - issuing a warning at or before time of tornado). Considerable improvements have been made in the last decade in that regard (POD has increased from around 0.6 to 0.7, which may sound small but it's really not). False alarms (warnings that don't verify) have stayed about the same during this period (note, most recent data for this is from 2004) which does suggest an increase in forecasting skill.

For mets, there was a nice article from Harold Brooks on this in BAMS 2004.

Back to the main topic, I think Tim and others identified many of the key players in hyper-sensitivity, though another may be the pressure of a future where automated forecasts (generated from models only) replace human forecasters. This is almost certainly a question of 'when' moreso than 'if', threatening the local office up to the national centers. Pressure on the humnan forecaster to remain relavent is ever increasing, while at the same time they are tasked more and more products that consume their time.