Accuracy of watch probabilities

[Mike Smith]

Well, I've just about exhausted my thoughts on this topic. Thanks, everyone!

 

[Greg Stumpf]

Well, I've just about exhausted my thoughts on this topic. Thanks, everyone!

..which is also why you haven't heard from me in a few days - plus, it is a holiday weekend!

But, I must comment on this:

Go to: http://www.norman.noaa.gov/nsww/talk...fri_stumpf.pdf which is Greg's powerpoint presentation on probability warnings. Starting at slide 9, overlaying probabilities onto polygons is exactly what he is depicting.

It is clear to me now that you were either not listening at that point of my talk or you've highlighted a need for me to do a better job at communicating this concept (although many others who have seen this talk have understood it, such as Rob and Mike G.).

That is not what I was depicting nor describing by those series of slides. I was showing a scenario in which a user, if so inclined, could drill down into the legacy polygon warning product and reveal greater and greater information detail about the hazard threat. Not only that, it revealed more information about the downstream threat about the hazard with greater lead time, information that was excluded from the legacy warning polygon because it fell below the uncertainty threshold for that type of warning product.

I'm taking another break from this discussion, but I am reading it. When I have the time and energy to respond to some of the points raised, I will.

Also, we've talked in person about a visit to WeatherData sometime. I'd love too when my busy schedule permits, but Feb 11 is out for me unfortunately. I'll contact you privately about this someday.

 

[Kiel Ortega]

To clarify what us "mathheads" have been doing all "isolated" in Norman: since 2007 we've been bringing in forecasters from all over the country to give us feedback on a number of new warning decision making technologies and techniques, including probabilistic warnings. The heaviest years we worked on probabilistic warnings was in '07 and '08. In 2008 we hosted two workshops bringing in folks of all sorts of backgrounds (local EM to FEMA to social scientists to NWS folks to private sector to the researchers) where they were allowed to comment on a number of issues, including probabilistic warnings. Further, members of this isolated, mathhead Norman community have traveled outside of Norman, particularly to a Nowcasting workshop in Canada which brought forecasters from around the world to speak about their warning methods (some of which have already adopted certain aspects of what's been pitched around the US as probabilistic warnings). Just because someone from Norman hasn't visited a certain company or talked to a specific EM doesn't mean we're just running numbers and looking to muck things up because we have the mathematical know-how.

Second, there is no probabilistic warning system. There's an idea to have probabilities...in some fashion...somewhere for people to look at---IF THEY WANT. For something that doesn't exist, lots of discussion and push back. The steps that have been taken are initial steps towards a goal that I would argue isn't necessarily a probabilistic system. It's to assist forecasters in warning situations. There is a firehose of information flying (or about to) at forecasters. Dual-pol will allow for increased interrogation of storms; 3x the current base outputs with who knows how many derived fields. Social media proliferation for storm reporting. Let's add local modelling efforts and new efforts to do OBAN on current radar volumes to produce an analysis. There's only so much a forecaster can do with a storm...let alone storms. As someone who has been in the mathhead, isolated Norman community, I always saw our effort to primarily assisting forecasters who could be overloaded with information. At the same time, I think it was quickly realized that such information could be beneficial to end users.

 

[Rich Thompson]

I appreciate the attempts by Mike G., Rob D., and Kiel O., Bobby P. and Greg S. to explain what we mean by "probabilistic warnings". The basic confusion and resistance seems to stem from the notion that Joe Public won't understand probabilities, or will misuse the probs because the values appear to be low. Their understanding is certainly a concern, but the public are not the direct audience for the probs - the weather savvy folks *are* the direct audience who can then translate the info into any format they want *for* the public. It's a certainty that any future NWS warnings will contain something almost identical to the current storm-based warnings, except maybe there will be different rules for when to initiate a "new" polygon for a new county based on the probs.

This is not an attempt to break the current watch/ warning program, it's an attempt to provide more information. The example of the man in Greensburg illustrates part of the problem - why did he *never* take shelter until that evening? It wasn't just the tornado warning that spurred his action, it was the tornado bearing down on him that got him to act! Most of the public don't run and hide when a tornado watch or warning are issued, and they shouldn't. Instead, these NWS products are meant to give everyone a fighting chance against potentially adverse impacts of severe storms. If we (the NWS) can get your attention (through the media), then we've done our job. We can't tell you exactly when to act or what to do because those decisions depend on circumstances we can't possibly know for each person. So, our approach is to provide enough information for you to make informed decisions. Probabilities are just another step in that direction.

p.s. The whole notion of the guy not responding to a "70% chance of a tornado in Greensburg" assumes that he's expecting higher numbers and has no additional experience with warnings. The probs for a Greensburg event would almost certainly be much higher than anything he'd seen before for Greensburg, which is an example of how the tornado warning can be enhanced. Plus, I doubt this guy would ever see those numbers directly - he might see a shaded graphic on TV with something like a "high" risk for a tornado which includes Greensburg.

p.p.s. False alarms do cause problems with complacency in the system. Wouldn't you rather see an honest meteorological assessment of the expected outcome (with uncertainty included), as opposed to a "black box" categorical warning? The same reasoning applies to the convective outlooks and watches, where the probs *augment* what's already produced and help ensure more consistency from forecaster to forecaster.

 

[Mike Smith]

p.s. The whole notion of the guy not responding to a "70% chance of a tornado in Greensburg" assumes that he's expecting higher numbers and has no additional experience with warnings. The probs for a Greensburg event would almost certainly be much higher than anything he'd seen before for Greensburg, which is an example of how the tornado warning can be enhanced.

I really didn't want to write any more but feel I need to respond to the comment above. In most parts of the United States a tornado warning is a rare or infrequent event. Lets say the NWS issued a tornado warning for Worcester County, Mass. (which is actually the city of Worcester) for an approaching tornado. While a citizen of Cleveland Co., OK or Sedgwick Co., KS might have enough experience with warnings to think with 20% "this is a high number" I virtually guarantee the citizen of Worcester would not.

Finally, the comments above keep asserting the present system will not be replaced. That is contrary to the AMS document, published just last week. Thus my concern.

 

[Mike Gibson]

I really didn't want to write any more but feel I need to respond to the comment above. In most parts of the United States a tornado warning is a rare or infrequent event. Lets say the NWS issued a tornado warning for Worcester County, Mass. (which is actually the city of Worcester) for an approaching tornado. While a citizen of Cleveland Co., OK or Sedgwick Co., KS might have enough experience with warnings to think with 20% "this is a high number" I virtually guarantee the citizen of Worcester would not.

Probabilities are weird things and hard to relate to, imho. It's all in the presentation. Your example of the Greensburg man favoring the 30% side is a good example. He's biased to the "no problem/do nothing" side because being hit by a tornado, historically, is extremely rare. In your discussions with people, you should turn it around and ask your audience if they would voluntarily do something given the same risk of disaster. For example, ask the Greensburg man if he would still get on an airliner if you told him shortly before boarding that there was a 70% chance the jet would crash on the way to Miami. My guess is he would say, "not only no, but hell no!". I'd bet that you could cut the odds way down, e.g. "1 out of 3 of the flights to Miami crash", and still get the same answer.

Finally, the comments above keep asserting the present system will not be replaced. That is contrary to the AMS document, published just last week. Thus my concern.

I think you misinterpreted what the AMS doc was saying. Several of the people actually designing/implementing the probability system, i.e. straight from the horse's mouth, have said that legacy warnings will be continued. What other reassurance do you need?

Mike

 

[rdale]

While a citizen of Cleveland Co., OK or Sedgwick Co., KS might have enough experience with warnings to think with 20%

Citizens aren't going to get the number, they're going to get the shaded in "higher risk" graphic on TV.

That is contrary to the AMS document, published just last week. Thus my concern.

Rich and Greg and Kiel and Bobby and the rest of the Normal mathheads are the people who would be presenting this to the NWS for implementation. They have made it clear that replacement is not an option - but enhancement is.

Why you put so much weight on an organization that has so little impact on the watch and warning system is confusing to me...

 

[Dave Kaplow]

I really didn't want to write any more but feel I need to respond to the comment above. In most parts of the United States a tornado warning is a rare or infrequent event. Lets say the NWS issued a tornado warning for Worcester County, Mass. (which is actually the city of Worcester) for an approaching tornado. While a citizen of Cleveland Co., OK or Sedgwick Co., KS might have enough experience with warnings to think with 20% "this is a high number" I virtually guarantee the citizen of Worcester would not.

Finally, the comments above keep asserting the present system will not be replaced. That is contrary to the AMS document, published just last week. Thus my concern.

As a lifelong resident of the Northeast, I don't think you are quite hitting it on the head in regards to the way people out here react to tornado warnings. While it's no doubt true that the average citizen of Worcester has little or no experience with tornado warnings, the very fact of their rarity means that most people will take warnings seriously when one is issued. Trust me, you start talking about tornadoes around here and people take notice, regardless of the details. For instance, whenever there's a tornado warning anywhere in CT, it's almost a sure bet that one or more of my friends, knowing my interest, will call and say frantically, "hey Dave, I just heard there's a tornado in... !" The word tornado just plain gets people's attention, and I don't really think any attached probabilities are going to affect that reaction one way or another.

If anything, I think the probabilities would help the situation here in CT, because the OKX office occasionally seems to be a bit too quick to issue warnings. I've seen them put out doppler-indicated warnings on clearly outflow-dominant cells that show, as far as I can see, virtually no signs of rotation on radar. Presumably they are acting on just about any transient TVS that pops up, in the belief that the high population density of the area justifies a better-safe-than-sorry attitude when it comes to tornado warnings. They may be right about that, but I would much rather see those type of warnings accompanied by a low probability number, so that the TV guys can tell the public something like, "the NWS has said that a tornado is possible with this cell, but they're not all that confident, so keep an eye out but don't head for the cellar just yet..." This would be far preferable to the current situation where the media gets all exited about cells that clearly have almost zero chance of tornadoing.

The bottom line is that I have to agree with those who maintain that the added information can only be of help, and I believe that even if the probability data does somehow get into the final warnings to the public, no one around here is about to ignore a tornado warning, not unless it's accompanied by precisely the kind of comment I have envisioned above. This kind of additional information can only help, whether one lives in tornado alley or a thousand miles away from it.

 

[Mike Smith]

Why you put so much weight on an organization that has so little impact on the watch and warning system is confusing to me...

Because there are at least 12 NWS people, some high level, whose names are on the document.

By the way, here is an example of operational probability forecasting: http://news.bbc.co.uk/2/hi/science/nature/8462890.stm

Mike

 

[rdale]

Because there are at least 12 NWS people, some high level, whose names are on the document.

I'm not sure if you ever participated in a commission before, but having your name in the appendix doesn't mean you support every word in every sentence of the commission's report.

I also don't see the head of the NWS listed there.

So 1) the people who would develop probability warnings say they will not replace the current system and 2) the NWS administration has made no claims to replace the current system with probabilities.

I'll avoid probabilities and say this categorically - probability warnings will NOT replace the current warning system.