Accuracy of watch probabilities

[Mike Smith]

Hi Everyone,

I am just back in my WeatherData office after speaking to the Wichita Kiwanis Club. During the Q&A session, one of the members asked this question, "I understand about tornado warnings being effective, but what about winter storms? There are winter storm warnings, heavy snow warnings, advisories, I get confused."

I said that I agreed with him about winter weather products (yes, I know the NWS cut them down in number 15 months ago) and said, "I have been engaged in a discussion the last few days regarding additional storm warning products centered around probabilities." The entire room groaned, loudly.

With me, was the non-meteorologist marketing director of Mike Smith Enterprises, the company I have formed for my book and professional speeches. Kim, take it away...

Mike is exactly right. More probabilities went over like a lead balloon, with groans and laughs from the audience.

...thank you, Kim.

I am out amongst the "public" (non-meteorologists, EM, etc.). They usually (not always) dislike the PoPs. I am nearly certain the general public will look at probabilistic watch/warning products with disdain.

We have to be on guard against our natural tendency to give customers what we think they should have versus what they want to receive and use.

 

[rdale]

I showed 'the public' a Powerpoint that Greg developed about probability convective warnings, and it was well received. So saying "We'll add more probabilities" and showing potential uses of probabilities should be considered two VERY different things...

 

[Mike Gibson]

I showed 'the public' a Powerpoint that Greg developed about probability convective warnings, and it was well received. So saying "We'll add more probabilities" and showing potential uses of probabilities should be considered two VERY different things...

Exactly. I also think it's important to note that the "audience" for the SPC's products are almost always computer programs now, not people. Those programs turn around and display the products for end users, e.g. a webpage of national or local graphics, localized text messages, etc. This frees the SPC to send out more raw data, more frequently, and not worry (or not worry as much) about the presentation to end users.

Mike

 

[Greg Stumpf]

I am just back in my WeatherData office after speaking to the Wichita Kiwanis Club. During the Q&A session, one of the members asked this question, "I understand about tornado warnings being effective, but what about winter storms? There are winter storm warnings, heavy snow warnings, advisories, I get confused."

I said that I agreed with him about winter weather products (yes, I know the NWS cut them down in number 15 months ago) and said, "I have been engaged in a discussion the last few days regarding additional storm warning products centered around probabilities." The entire room groaned, loudly.

Tell the same audience that the NWS issues most tornado warnings today with 30 minute durations. Then ask the audience if they would rather have longer lead-time hazard information, which could be useful when they are in situations where they need more than 30 minutes to take protective action (provide situational examples). You will get some yes answers, I can guarantee that. Then tell the audience that if longer lead-time tornado warnings were issued, they would have less accuracy than a 30-minute lead time warning. Then ask them if these longer lead-time warnings would still be useful to them in some way. And then explain uncertainty, and ask them how might forecasters communicate that uncertainty in the warnings with varying lead-times. Then repeat this exercise with a large number of groups of people to get a representative and statistically-significant sample.

Then ask meteorologists and developers what might be useful ways to create the hazard information such that it can be turned into effective products that meet this customer demand. One such way would be to produce digital gridded probabilistic hazard information, from which you can create a number of different types of products and services to meet the demand.

The bottom line - if you frame the problem in different ways, you might find that demand. No, I'll go out on a limb and say that you will find the demand.

(The above represents one possible method of how such a study could be accomplished, but has not yet been completely vetted by social scientists, decision scientists, or other experts in these multi-disciplinary fields.)

 

[Mike Smith]

Greg, in NO WAY did I bring the topic of "confusion" up, it sprung up from audience questions which led to the discussion about probabilities. I had Kim post because I wanted you to hear it from someone who is a non-meteorologist and with a marketing background. The thought of any more probabilities was a non-starter to this group.

Tell the same audience that the NWS issues most tornado warnings today with 30 minute durations. Then ask the audience if they would rather have longer lead-time hazard information, which could be useful when they are in situations where they need more than 30 minutes to take protective action (provide situational examples). You will get some yes answers, I can guarantee that. Then tell the audience that if longer lead-time tornado warnings were issued, they would have less accuracy than a 30-minute lead time warning. Then ask them if these longer lead-time warnings would still be useful to them in some way. And then explain uncertainty, and ask them how might forecasters communicate that uncertainty in the warnings with varying lead-times. Then repeat this exercise with a large number of groups of people to get a representative and statistically-significant sample.

Have been an expert witness numerous times (and conducted seminars for other expert witnesses), the technique you describe above is called "leading the witness." It is objected to in legal (and marketing) circles because it leads people to say what the person leading the discussion wants rather than the objective truth. I hope that doesn't sound harsh, it is not intended to be, but what I just stated is accurate.

Yes, when you lead people through a process like the one you describe, they will say they 'want' probabilities but that outcome is not necessarily indicative of their true feelings. A better process would be to sit people around a table and ask them an open-ended question along the lines of, "tell me what you like and don't like about weather forecasts."

I had not planned on posting any more on this subject and just as I had not planned on any of this coming up at the Kiwanis Club meeting, but it did. The audience's reaction (as witnessed by Kim and me) was so negative and derisive, I felt I should pass it along. I genuinely wish to be helpful to the NWS when it comes to public warning products.

Mike

 

[Greg Stumpf]

Yes, when you lead people through a process like the one you describe, they will say they 'want' probabilities but that outcome is not necessarily indicative of their true feelings.

My example above was designed to elicit statistical social data from users on how forecasters could best express uncertainty in long lead time warnings. I never said this was designed to lead the audience into "wanting probabilities" (those are your words).

The only thing I might be "leading" (if you will) the audience is to reveal that they would want warnings with longer lead times and then couch that with a discussion of the present state of the science.

I will repeat, for that last time, that:

• the users never ever have to see the probability numbers (unless they want to)
• we aren't replacing the present system of products
• we are looking for additional ways to provide hazard information that is more adaptable to the spectrum of users versus one-size-fits-all products

8 people died in Lone Grove OK, 4 people died at Little Sioux, other people died in their rural manufactured housing on Super Tuesday 2008, etc., etc. Countless others were injured (a statistic which is rarely used to measure trends in performance). Other users have said that the present suite of products doesn't express more detail about the timing and location of threats. We can improve.

 

[rdale]

The audience's reaction (as witnessed by Kim and me) was so negative and derisive, I felt I should pass it along.

I think if you SHOW them probabilistic warnings - the reaction will be dramatically different.

 

[Mike Smith]

8 people died in Lone Grove OK, 4 people died at Little Sioux, other people died in their rural manufactured housing on Super Tuesday 2008, etc., etc. Countless others were injured (a statistic which is rarely used to measure trends in performance). Other users have said that the present suite of products doesn't express more detail about the timing and location of threats. We can improve.

I hate to keep this thread going because I don't want to create any antagonistic feelings, but I must point out the following:

If we measure deaths per million population from tornadoes from before there was a tornado warning system to the mean of the last three years (2006-2009), we have cut the tornado death rate by 98%!!!

That is an amazing scientific accomplishment about which the entire meteorological profession should take tremendous pride. And, we have done it at a very low cost to society.

At the December, 2008 Warning Workshop in Norman, I about spit out my lunch when one of the people at my table opined that the warning system was a "failure" (his exact word). I said, "on what do you base that?" He said, "Nine people died at Greensburg." I have studied those unfortunate situations. One death was a truck driver who was in the wrong place at the wrong time. We could probably have helped him with better technology. The other eight were all in shelter. One unfortunate woman died when a highway guard rail from U.S. 54 was thrown through the roof of her home, through her floor and impaled her in the chest. How would probabilities have helped her?

Given that eight of the nine who died were in shelter, in what way does that make the warning system a "failure." How do you improve on 99+% of the population in shelter??!! And, they were inspired to go to shelter by the existing system.

No matter what we do, people are going to die in tornadoes, especially F4's and F5's. I know that sounds cold, but it is a fact.

Now, I would be the last person to say we should not try to improve. But citing these unfortunate deaths without the greater context that only 22 people died nationally last year, to me, misstates the nature of the challenge of how do we improve a hugely successful system? The reason I don't see probabilities as something we should spend much time on is that I believe (as in Lone Grove) a better system to communicate the existing warnings is likely to give us "more bang for the buck" (given finite resources) than probabilities.

OK, I'll stop posting. Thanks everyone, and especially Greg and Rich, for the insightful and respectful dialogue.

 

[rdale]

The reason I don't see probabilities as something we should spend much time on is that I believe (as in Lone Grove) a better system to communicate the existing warnings is likely to give us "more bang for the buck" (given finite resources) than probabilities.

Why not do both? Most forecasters already have probabilities in their mind when they issue the warning, so put it in a format that others can use? If you look at Greg's work - he's not proposing that you start drawing the 70% line on the air and tell people they have a 73.425% chance of a tornado, but the TV met converts the probs to the "old fashioned" warning.

Then everyone gets more bang for their bucks...

 

[Mike Johnston]

Goodness gracious, I thought we were talking about the utility of using probabilities with watches, and now we're talking about using them in warnings?

Personally, in the context of warnings - on the scale of minutes - I don't want one forecaster spending even a few seconds attempting to convert anything into probabilities. At that point, the sole focus should be on accuracy, to the best of their ability. Similarly, I don't want a TV met spending even a second of effort to convert any probability information into a "traditional" warning communication. What is the practical benefit at that point? What are the grid box sizes at the warning level and does it even matter in the real world? At a certain point in space and time, messing around with probabilities must have a diminishing return in terms of the practical benefit of the forecast. Let's not confuse the real mission with theoretical absurdity.

I mean, an actual warning either verifies or it doesn't. At that point, what interests are really served by assigning a probability (other than the warning forecaster's CYA to hedge his own stats)? I wonder what Fawbush and Miller would have thought about this whole discussion?

 

[rdale]

Personally, in the context of warnings - on the scale of minutes - I don't want one forecaster spending even a few seconds attempting to convert anything into probabilities.

If they are good - they already do. When you see the storm near the edge of the warning, the good ones say "If you're in xxxville, you have no warning yet but I'd start thinking about taking cover." That's what the "moving" warning box does so that Joe xxxville Citizen can get that info without watching the TV. And if you know anything about local TV newscasts, you can bet that the number of good meteorologists on TV will be MUCH lower in 10 years than it is today.

I wonder what Fawbush and Miller would have thought about this whole discussion?

They would have been impressed with how far we've come. Have the people instantly rejecting probability warnings looked at any of Greg's presentations to see how it might be used?

 

[rdale]

This just in...

---

The AMS Ad-Hoc Committee on Uncertainty in Forecasts (ACUF) was charged by the AMS Commission on the Weather and Climate Enterprise Board on Enterprise Communication to engage the weather, water, and climate community in formulating a cross-enterprise plan to provide forecast uncertainty information to the Nation. The current draft of their plan is now available for viewing and download at:

[URL]http://www.ametsoc.org/boardpges/cwce/docs/BEC/ACUF/2010-01-Plan.pdf

[/URL]The Committee will host a Town Hall at the AMS Annual meeting next week on Tuesday, January 19, at noon in Convention Center Room B212 to discuss the plan. They are also asking for those interested to review the plan and send comments and suggestions to: [email protected] by January 31, 2010.

Paul Hirschberg
Co-Chair, Ad Hoc AMS Committee on Uncertainty in Forecasts

 

[Mike Gibson]

Personally, in the context of warnings - on the scale of minutes - I don't want one forecaster spending even a few seconds attempting to convert anything into probabilities. At that point, the sole focus should be on accuracy, to the best of their ability.

You're already seeing probabilities at work in the warning, except that you're viewing the "categorical" presentation of them -- the "warning" category. The NWS met, after seeing the radar data, receiving reports, etc. decided that the probability of a tornado was high enough for a warning. He may have been watching that storm for 30 minutes already but wasn't confident enough, i.e. the probability was too low, to issue a warning.

Also, if you like the new "storm-based" warning outlines, then you like the probability approach. Why does the warning outline widen as you get farther away from the initial position? Because the probability of the tornado being in a particular location farther out in time decreases (the forecast position is uncertain).

What Stumpf and the others want to do is send out these probabilities in a rapidly+automatically updating, high resolution grid. The local NWS met decides that there's a high probability of a tornado at a specific point. He notes that point, probability, and the projected storm motion. The software sends that info to the national center which fills in the affected points in the national grid with probabilities that are highest along the projected path and fade as you get farther away from the initial position (in both space and time). The grid automatically updates (say, every minute), meaning the probabilities grow and fade along the projected track over time.

What's nice about this is that tornado probabilities due to a warning in Cherokee county to my west automatically move towards my county over time. I don't go from "no warning" to "tornado warning" in a single step when a new warning is issed, rather the probability at my location increases over time. In the software I'm using to view those probabilities and convert them to categories, I could set it to give me a warning for 5% prob at 60 minutes, 10% prob at 30 minutes, etc. (I'm just tossing out numbers here, the SPC/NSSL would do the research to give guidance on the potential values). Alternatively, I could set the software to an SPC-like risk category presentation: none, low, moderate, high, extreme.

It's an ambitious plan and the implementation is nontrivial but I imagine after a few years of usage you'll wonder why it wasn't done a long time ago. I'd like to see NOAA start with a less difficult task and give this same treatment to the watch timeframe but on a lower resolution grid -- a semi-automated "Convective Outlook". The SPC experts would periodically do their analysis to produce an overall probability outlook and then high resolution models would fine tune those probabilities automatically over time as the situation unfolds. This is similar to the SWODY1 and associated MCDs, but in a grid format that updates more rapidly.

Mike

 

[rdale]

The presentation from last year's OUN conference is at http://www.norman.noaa.gov/nsww/talks/007_fri_stumpf.pdf

 

[Mike Smith]

Have the people instantly rejecting probability warnings looked at any of Greg's presentations to see how it might be used?

I really didn't want to post again, but I must.

Yes, I have seen Greg's presentation. I contend the constantly updated "probability warnings" have the potential to really screw up the hugely successful binary (warning/no warning) warning system, at least as Greg presented his concept proposal in Norman in December, 2008 (link to presentation above).

The presentation and its graphics are based on the premise of a single supercell in the warning area with the on-air meteorologist portraying the ever-changing path and probability projections. Seems simple, right?

But that is not how the real-world of severe weather works. The single well-behaved supercell in a television viewing area is a rarity. The far more common situation is a multi-storm situation. In the case of a severe weather outbreak (when warnings are at their most important), the ever-changing probabilities and path projections will create a level of complexity that will challenge the TV meteorologist to handle and be virtually impossible for the viewers (the public) to understand.

Lets take a recent example, the May 8, 2009 vicious derecho that started in the ICT CWA and violently made its way into Kentucky. Here is the radar image of the derecho over southwest Missouri

and a map of the tornadoes and straight winds featuring tornadoes of F3 intensity

Now, try to envision a probability of tornado, probability of damaging winds (and straight winds of more than 100 mph were measured about the time of the radar image), and probability of large hail (and 2 3/4 inch hail, driven by 80+ mph winds were reported) all on a television screen, constantly updated with overlapping probability tracks.

Because of the rapid movement of these storms, a viewer in Carthage, MO would be seeing at least two sets of TOR, WIND, and HAIL probabilities for both both the approaching supercell (a minute or two away) and the main derecho. The level of visual and numeric complexity and thus potential for confusion due to "information overload" is HUGE. When you add in that somehow people are supposed to keep in mind that 30% equals a high threat level, I just can't see this working in the real world.

What seems straightforward in theory often isn't in practice.