Accuracy of watch probabilities

[Randy Bowers]

From a post by Wes Carter in response to Greg's comment about 2/5/08 that was apparently removed:

"... Many people were tuned into CNN, MSNBC, Fox, and national news on their satellite dishes and cable rather than to the local TV stations..."

I challenge anyone to offer an explanation as to how probabilistic warnings will address the broader issue of folks not getting the warning. Not to stray off topic, but I think it's necessary to include this, the #1 reason people die from tornadoes, in the discussion, and to at least give that consideration when developing new warning techniques.

I would like to pose the following question. Where is the money, time, and effort best spent if the objective truly is saving lives?

A.) Significant adjustments to a warning method that's already been proven amazingly effective (as Mike Smith has pointed out).

B.) A stepped-up effort to address the issue of folks not receiving the warnings through programs such as WAS*IS, etc.

Personally, I choose B.

I agree with Mike on the issue of probabilistic warnings, although I seem to be the minority in this crowd.

Having said that, I will give support to Rich Thompson and the SPC staff for probabilistic forecasts and watches. I do believe these will provide great value from a guidance standpoint to the meteorology, emergency management, and broadcast communities, although I'm somewhat reluctant to agree with the notion that the public in general will directly benefit from them. As was mentioned previously, those who understand probabilities and are comfortable with using them will most certainly benefit. The rest would likely benefit indirectly.

 

[Mike Gibson]

Mike, read the section of the AMS report I quoted: "Shifting" to probabilistic warnings. That is their word, not mine.

It's my understanding that even if the underlying warning system were switched to probability grids, the NWS would still derive traditional warnings from them for public dissemination. Have you heard different?

The "shifting" phraseology was also used in the breakout groups at the warnings conference in Norman in December, 2008. What was really interesting about that is there were two TV mets, two private sector mets, two EM's and two social scientists and every member of that group was against probability warnings. Each member of the Norman subgroup was in favor.

Nothing earthshattering there. It's a given that the core designers will be in favor of something new well before the end users. Happens all the time in software.

We have a system that has been amazingly successful and saves 98% of the potential deaths from tornadoes. Why do we want to "fix" a system that is obviously not broken?!

I don't believe this new system's sole purpose is reducing deaths from tornadoes. Did you have the same objections to the introduction of storm-based warnings?

Mike

 

[rdale]

A.) Significant adjustments to a warning method that's already been proven amazingly effective (as Mike Smith has pointed out).

B.) A stepped-up effort to address the issue of folks not receiving the warnings through programs such as WAS*IS, etc.

Why are those mutually exclusive? Dual-pol radar is on the way... Should we stop development on those projects and instead give all the funds to option B? I can't imagine anything other than the HCA product showing up in the public eye - so should this be considered a waste of money because most of the products benefit meteorologists / EMA / etc.?

I certainly agree that societal response issues need to be addressed. But I disagree that we need to stop development of new tools to issue those warnings in the first place.

 

[Mike Smith]

It's my understanding that even if the underlying warning system were switched to probability grids, the NWS would still derive traditional warnings from them for public dissemination. Have you heard different?

My Merriam-Webster dictionary says the meaning of shift (in that context) means, "to replace with another." That says to me they want to replace the current system with a probability-based system. If you read the AMS document, that is clearly the intent.

And, yes, I have heard the "shift" language from others. That is why I am spending all this time on this.

I don't believe this new system's sole purpose is reducing deaths from tornadoes. Did you have the same objections to the introduction of storm-based warnings?

Mike

If the purpose is not to reduce deaths, what is the point? Remember, we are talking about public warnings.

I was one of the strongest proponents of storm-based warnings because they improve the accuracy of the current system that people know and understand.

I'm going to say it again: We are weather enthusiasts and this is important to us. The vast, vast majority of the population believes storm warnings are a necessary nuisance. They feel slightly silly coming out of the basement after a false alarm. Telling them (to use Bobby's example), there is a 10% chance of a tornado 30 minutes in advance rather than TORNADO WARNING, TAKE COVER! will provide a tremendous reason to rationalize not taking cover.

In my Greensburg research, I learned that, for one 88-year old survivor, it was the first time in his life he had taken cover (not going to the basement was a point of pride with him). Via his son I learned the reason he went to the basement this time was the combination of the siren, the TV tornado warning, and the live reports of the tornado doing damage to the south of town. Because of power failures ahead of the tornado, we know that the last of the information everyone received (i.e., before power started going out) was at 9:43. Because this is 15 minutes before, Chuck's, "I would have told them 70% fifteen minutes before the tornado" would probably have meant that this man would have sat the storm out like he had every other in his life. And, that might have meant his eventual demise.

Mike

 

[Mike Gibson]

My Merriam-Webster dictionary says the meaning of shift (in that context) means, "to replace with another." That says to me they want to replace the current system with a probability-based system. If you read the AMS document, that is clearly the intent.

And, yes, I have heard the "shift" language from others. That is why I am spending all this time on this.

I'll let the folks actually planning the system clear up this misunderstanding.

If the purpose is not to reduce deaths, what is the point? Remember, we are talking about public warnings. I was one of the strongest proponents of storm-based warnings because they improve the accuracy of the current system that people know and understand.

How did storm-based warnings, which you strongly supported, reduce deaths? What was the point of making that change? If anything, reducing the coverage of the warned area could result in an increase in deaths because fewer people are included in the warned area and storm motion/development/etc. is uncertain.

Mike

 

[Mike Smith]

I'll let the folks actually planning the system clear up this misunderstanding.

What misunderstanding? The document, published just last week, says what it says.

How did storm-based warnings, which you strongly supported, reduce deaths? What was the point of making that change? If anything, reducing the coverage of the warned area could result in an increase in deaths because fewer people are included in the warned area and storm motion/development/etc. is uncertain.
Mike

I think storm-based warnings are new enough we don't know the answer yet. I believe the importance of storm-based warnings will become clear over the next couple of years as technology catches up to the improved science.

My theory of how they reduce deaths is that, by reducing "false-alarm area," we increase the credibility of the warning system so that people respond when a warning is issued. We decrease the people who feel "slightly foolish" for taking cover. We reduce the toll on, for example, restaurant owners' revenue ( www.weatherdata.com/services/news_falsealarms.pdf ) due to false alarms -- which means they are not lobbying against new sirens.

Remember, we are (thank goodness!!) dealing with a much smaller population of tornado fatalities than 50 years ago. So, finding out to what extent storm-based warnings are helping is going to take longer and may have a less pronounced "signal to noise" ratio than earlier warning improvements. It is more difficult to save that last 2% than it was the first 98%.

For those who have not read this entire thread, please go back and look at my post #60. No one has responded as to how they would have conveyed the current warnings plus superimposing probability warnings of tornadoes over that critical, life-threatening situation without confusion or "information overload" in real time with (as some probability advocates suggest) updates of radar, paths and probabilities every five minutes.

One strength of the current warning system is that radio + TV + siren are all saying the same thing: TORNADO WARNING, TAKE COVER! I believe Eve Gruntfest and other social scientists will say that it is important for each source of the warning to reinforce the others. I'll say it again: How are radio and siren going to convey a "probability"? If that radio is saying TAKE COVER and the TV is showing 30% chance of a tornado, why doesn't that cause confusion?

Mike

 

[Randy Bowers]

Dual-pol radar is on the way... Should we stop development on those projects and instead give all the funds to option B?

I'm not advocating the halting of dual-pol at all. Dual-pol has great potential to offer warning decision makers more confidence in their warning decisions and that's something that we absolutely need to continue to go forward with. My point is that careful considerations must be given when devoting resources and funding toward a revamping of an already very effective warning system. Dual-pol and other future technologies that will aid WDM's in their decisions are much different that technologies such as probabilistic warnings which add nothing to the WDM's confidence in there being a tornado or not.

To answer this question directly, I really can't say since I'm not in a position to know how much funding is being allocated and to where. The point I was trying to make is that I would hope the primary focus with regards to funding and resources is weighted toward the societal issues at this point since I think most of us would agree that the overwhelming majority of tornado deaths are due to the societal issues that have been discussed on Stormtrack and elsewhere quite extensively.

I can't imagine anything other than the HCA product showing up in the public eye - so should this be considered a waste of money because most of the products benefit meteorologists / EMA / etc.?

You have a reasonable good argument here. But, how high is our confidence that this would ultimately meet our objective of saving more lives? I have yet to see sufficient evidence that inadequacies in the current warning system (binary / storm-based) are resulting in deaths.

 

[Bobby Prentice]

Bobby,

You, Rich and Greg write as if these products are not being created and delivered now. They are by private sector weather companies. The fact the NWS doesn't do something, doesn't mean it isn't done.

Mike,
Sorry if what I wrote sounded as though there were no probabilistic forecasts being generated by private weather companies. Of course, I know they are done. My main purpose was to educate others on this list who may be unaware of the use and potential value of probabilistic forecast products. It goes back to my natural inclination as a meteorologist instructor.

I really believe (and I mean this in a constructive, encouraging manner) that you guys need to get out of Norman more and see what the rest of the field is doing. You'd be amazed what we do and how advanced we are!

I can only speak for myself, but yes, I have been out of Norman before. I've worked in other states, meet with weather customers on a daily basis, attend user group conferences, toured various private weather companies, etc.

I'll make you and any of your Norman NWS/OU/NSSL colleagues an offer: Come up February 11 for our AMS luncheon meeting and I'll give you the grand tour of WeatherData after. The meeting location, Sedgwick Co. Emergency Management, and WeatherData are just a few blocks from each other.

Unfortunately, my work and family considerations won't allow me to attend this event. However, I'd love to visit your operations at WeatherData. Perhaps I could visit some time in the future (like during my chase vacation)?

I am very familiar with the Wichita-Andover close call with regard to the B-1's and KC-135's at IAB. With all due respect, I believe you have the approach backward: No base commander is going to scramble the aircraft with a 5% probability and, by the time it goes above 10%, it is too late (given your scenario). I quote the late Col. Bob Miller in my book on this topic. Scrambling expensive aircraft, given a 5% chance, would lead to the base commander being "laughed out of this man's Air Force."

A 5% chance of the base getting hit by a tornado with billions of dollars of mobile assets at stake and action which could be done to save the aircraft is not taken? Couldn't they at least move the aircraft into hangers to alleviate the hail damage? The Air Force did this very thing at Tinker AFB in Oklahoma City during the early research days of Doppler weather radar. Saved multi-millions of dollars worth of damage. I don't believe the base commander got "laughed at" for that one.

I have never been against all uses of probabilities as a reading of this entire thread will reveal. I am against upending the current watch/warning system for the public. Telling people there is a 10% chance of a tornado in 30 minutes is a great way to convince them not to go to shelter. It is amazing to me that so many in Norman (and I am aware of others) want to fix a system that is most certainly is not broken.

...read the section of the AMS report I quoted: "Shifting" to probabilistic warnings. That is their word, not mine.

I can't speak for the AMS, but perhaps either they misstated or you misunderstood what they meant? The proposals I've seen have been to augment the current warnings with probabilities. The public includes some pretty sophisticated users too (like those on this list). It's easier to begin with a probabilistic watch/warning and then assign a probability value which would trigger the public binary product. Of course, this probability value should be assigned after careful research and consultation. Perhaps this is what they really meant?

At the request of the ICT AMS members, I am doing Miracle at Greensburg for that meeting. The Greensburg presentation has been viewed by the mayor of Greensburg, the head of the Kiowa County Commission, and 200+ citizens of Greensburg. It has been very well received and I have been praised for its accuracy. I sincerely believe you will be even prouder to be part of the meteorological profession once you see it.

I'm sorry I can't be there to see it. Perhaps you could video tape it and post it online?

 

[Mike Smith]

Unfortunately, my work and family considerations won't allow me to attend this event. However, I'd love to visit your operations at WeatherData. Perhaps I could visit some time in the future (like during my chase vacation)?

A 5% chance of the base getting hit by a tornado with billions of dollars of mobile assets at stake and action which could be done to save the aircraft is not taken? Couldn't they at least move the aircraft into hangers to alleviate the hail damage? The Air Force did this very thing at Tinker AFB in Oklahoma City during the early research days of Doppler weather radar. Saved multi-millions of dollars worth of damage. I don't believe the base commander got "laughed at" for that one.

I'm sorry I can't be there to see it. Perhaps you could video tape it and post it online?

Bobby,

Thanks for clarifying. Yes, you are welcome when you are in the area. The only thing I ask is you call first.

In addition to www.mikesmithenterprises.com, I am creating a second web site, www.warningsbook.com (now pointed to the book portion of my web site) that will have the video tape of Miracle at Greensburg and some terrific video material that (obviously) cannot be included in the book itself. The 'warnings' web site is intended for book purchasers. So, buy the book and watch the video.

With regard to the base commander. Assume, as in your original post,

A supercell thunderstorm forms 2 hours (T-120 minutes) southwest of of Wichita and the NWS issues a probabilistic tornado warning graphic for its path that indicates a 1 percent chance of a tornado hitting within 5 miles of the center of McConnell AFB. At T-90 minutes the warning is updated to a 2% chance. At T-60 minutes, a 5% chance. At T-30 minutes, a 10%. At T-5 minutes, a 99% chance.

In order to safely get the planes out the operation would have to start at the one hour mark. Assume well-calibrated probabilities and that the tornado (or large hail) occurs on the 16th occurrence. It is hard for me to imagine that after very expensively flying the aircraft to other bases 15 times that the base commander would be willing to fly them out on the 16th. I have observed this behavior in our customers. There was an electric utility who stopped using some very well-calibrated probabilities after the numbers went against them a couple of times.

Thanks for responding to my post.

Mike

 

[Mike Gibson]

What misunderstanding? The document, published just last week, says what it says.
...
For those who have not read this entire thread, please go back and look at my post #60. No one has responded as to how they would have conveyed the current warnings plus superimposing probability warnings of tornadoes over that critical, life-threatening situation without confusion or "information overload" in real time with (as some probability advocates suggest) updates of radar, paths and probabilities every five minutes.
...
I'll say it again: How are radio and siren going to convey a "probability"? If that radio is saying TAKE COVER and the TV is showing 30% chance of a tornado, why doesn't that cause confusion?

There's your misunderstanding. No one has suggested showing the raw probabilities to the public or overlaying raw probabilities over a tornado warning. There will still be legacy, binary warnings and their polygons. Many replies have pointed this out and I hope that one of the much-derided Norman folks will step in to reassure you of this fact.

I think storm-based warnings are new enough we don't know the answer yet. I believe the importance of storm-based warnings will become clear over the next couple of years as technology catches up to the improved science.

My theory of how they reduce deaths is that, by reducing "false-alarm area," we increase the credibility of the warning system so that people respond when a warning is issued. We decrease the people who feel "slightly foolish" for taking cover. We reduce the toll on, for example, restaurant owners' revenue ( www.weatherdata.com/services/news_falsealarms.pdf ) due to false alarms -- which means they are not lobbying against new sirens.

Warned areas derived from the probability grids will be more accurate than the simple polygons in current storm-based warnings, further reducing the false alarm syndrome. In addition, the probability areas will automatically move in time, eliminating the temporal "chunkiness" of the current warning system. Instead of polygonal warnings that are chained end to end where the first point in the path gets, say, a 15 minute lead time and one at the end has an hour lead time, all locations in the path of a storm will receive approximately the same lead time for an ongoing event. People who rely on the legacy representation of warnings will actually be at a disadvantage to those able to receive warnings derived from the probability grids.

All the reasons for your strong support of storm-based warnings are equally applicable to probability-derived warnings.

Mike

 

[Mike Smith]

No one has suggested showing the raw probabilities to the public or overlaying raw probabilities over a tornado warning.

In addition, the probability areas will automatically move in time, eliminating the temporal "chunkiness" of the current warning system. Instead of polygonal warnings that are chained end to end where the first point in the path gets, say, a 15 minute lead time and one at the end has an hour lead time, all locations in the path of a storm will receive approximately the same lead time for an ongoing event.

Go to: www.norman.noaa.gov/nsww/talks/007_fri_stumpf.pdf which is Greg's powerpoint presentation on probability warnings. Starting at slide 9, overlaying probabilities onto polygons is exactly what he is depicting.

Go to the AMS paper: www.ametsoc.org/boardpges/cwce/docs/BEC/ACUF/2010-01-Plan.pdf
lines 308 to 315, they are exactly advocating "shifting to probabilistic forecasts and a hazardous weather and warning capability..." There are at least 12 NWS names on the list of advocates in the Appendix.

How can you say, "No one has suggested showing the raw probabilities to the public or overlaying raw probabilities over a tornado warning."? That is exactly what both are saying!

I still want to see a "non-confusing, easy-to-understand at a glance" of polygons with probabilities in the mixed supercell/derecho situation (with the forward motion of the derecho 70+mph) in post #60 that can be used on television, by far the #1 source of storm warning information.

 

[rdale]

Starting at slide 9, overlaying probabilities onto polygons is exactly what he is depicting.

Of course. But not for the general public. See page 35 - it doesn't get much clearer than that.

they are exactly advocating "shifting to probabilistic forecasts and a hazardous weather and warning capability..."

Do you honestly believe that the AMS sets policy for the NWS and TV mets?

 

[Mike Smith]

Of course. But not for the general public. See page 35 - it doesn't get much clearer than that.

Except that is contradicted by slides about "will the public understand?". (slides 37-40)

Do you honestly believe that the AMS sets policy for the NWS and TV mets?

TV mets through its certification programs, yes. NWS, no.

 

[rdale]

Except that is contradicted by slides about "will the public understand?". (slides 37-40)

I don't think that's contradicting... Certainly SOME of the public will understand. I'd say 75%+ of all storm chasers are not meteorologists or EM's (e.g. "the public). Yet many of them understand the probabilities in DY1 outlooks.

TV mets through its certification programs, yes.

How'd that work with Dr Cullen? The AMS does not have the ability to control any TV mets, certified or not, without the backing of the mets. I'd suggest looking at the certification process as it's apparently changed since you last checked.

These days many mets have to send in sample tapes that never make the air. The AMS requires frontal analysis, climate data and more. Most TV mets don't show any of those, so the AMS accepts "demo tapes."

Let's say somehow the AMS decided that all seal applicants need to show probabilistic warnings. 1) All mets who already have their seal have no incentive to change. 2) All mets who want to get their seal will make a demo tape off-air with the probabilities.

End result? No change. The good-old county-based warning system is here for a LONG long time. Adding probabilities hurts nothing and helps many.

 

[Mike Gibson]

Go to: www.norman.noaa.gov/nsww/talks/007_fri_stumpf.pdf which is Greg's powerpoint presentation on probability warnings. Starting at slide 9, overlaying probabilities onto polygons is exactly what he is depicting.

Good grief. Stumpf's showing how the probabilities compare to current polygon warnings. Look at slide #10. The red area in the probability grid corresponds to the legacy binary warning derived from the grid. Look at slide #35, "How these gridded data can be used", showing the conversion from the probability grid to a legacy binary warning. Look at slide #42, "What are the best ways to convert gridded probabilistic hazard information to useful products?"

Go to the AMS paper: www.ametsoc.org/boardpges/cwce/docs/BEC/ACUF/2010-01-Plan.pdf

lines 308 to 315, they are exactly advocating "shifting to probabilistic forecasts and a hazardous weather and warning capability..." There are at least 12 NWS names on the list of advocates in the Appendix.

The important part: "which incorporates probabilistic forecasts and thresholds into the warning criteria". Probability + criteria = warning. Not Probability = Warning.

How can you say, "No one has suggested showing the raw probabilities to the public or overlaying raw probabilities over a tornado warning."? That is exactly what both are saying!

See above.

I still want to see a "non-confusing, easy-to-understand at a glance" of polygons with probabilities in the mixed supercell/derecho situation (with the forward motion of the derecho 70+mph) in post #60 that can be used on television, by far the #1 source of storm warning information.

You won't see that on TV unless the station configures their tornado/hail/etc. probability display software to show it that way. What you would see are red or purple areas and/or outlines showing where the warning thresholds have been met. Some may choose simply display the official legacy warnings from the NWS that were derived from the grids.

You and I may want to see the raw probabilities. Or where the 1hr threat is above xx%. Or where the threat has increased significantly in 1hr. Or... insert your desired graphical presentation method here.

We can't do any of that right now because the NWS hasn't made the raw probability data available. We can approximate it with our own homebrew schemes but I'd prefer to start with official NWS data (I've tinkered with a very primitive, low res attempt). I want to concentrate on the much easier presentation side of the problem.

Mike