Accuracy of watch probabilities

[Mike Smith]

I may have no hope of convincing you that probabilities are useful, but that doesn't matter. You're free to put all of your emphasis on the public watch products as they have existed for 40+ years (and will continue into the foreseeable future). The probs are nothing more than additional information that you can choose to consider or ignore.

Actually, am open to the concept of probabilities in certain circumstances, provided the probabilities are well-calibrated which is certainly (in my opinion) not the case now. So, they have the potential to do more harm than good.

If you had asked me all of this before I did my four year project of researching and evaluating the storm warning system for my forthcoming book, I would have likely tended to agree with you and Greg. I have learned that the current program is amazingly effective, more than most meteorologists believe. So, I tend to be cautious about things that could damage that success.

I conclude that -- for the near to medium term -- let the private sector handle the probabilistic values for specialized users and be very cautious about tinkering with the public products.

 

[rdale]

I need to know the specific date you're writing about for me to comment on the trends in the watch numbers for that day.

I don't remember the event offhand, I had a good discussion with Greg Carbin about the event and will see if I saved the emails.

I've dealt with the watch probabilities for a few years, and the tornado numbers are reliable. That means when we say 40%, it happens pretty close to 40% of the time.

The tornado numbers are verifying, or the F2+ tornado numbers are verifying? That's the eye catcher for me more than anything.

My current career is now emergency management. Given my met background, I'm certainly not the typical EM so I won't give my opinion on how I use the probabilities. But having dealt with many other EM's in the region, I don't know of one EMA that would do ONE thing different for a 20% tornado watch vs 60% tornado watch. If a tornado could hit their county, they need to be prepared. End of story. You could throw out a 80% chance of F2+, but does that mean I (a generic EM) notify hospitals or close schools or anything else? I don't think so - because if that F2 hits on the complete other side of the watch box then SPC verifies (good) but all the efforts I put into my county are busts (bad.)

{Side note, I think it's a horrible call for school admins to close down for a watch. That's the lawyers talking more than common sense.}

Tim is correct. There isn't ONE watch probability that all EMs / firefighters / school admins / etc will use as a cutoff. My question-- is there any probability that any of them would use to do anything different? I remain unconvinced. But if you can come up with an example, I'd love to hear it.

Now one area I am a big fan of is Greg's work with probability warnings. That's MUCH more tangible for an EM or fire department to utilize, and can be easily converted to "old school" style warnings for the public. But I don't think in an area we know so little about (3-6 hour small scale tornado forecasting) we should expect people to make possible life-changing decisions because a particular SPC forecaster put out 60% chance of F2's on that day.

- Rob

PS I've found that if you ask people (EMs, fire departments, etc) if they would like to see xx or yy in their forecasts or watches or warnings or whatever they'll say YES to anything. They always want more. Sometimes their requests are pretty stupid. If you do them, will they get upset if you take them away later? Yes. If you ask them how they are specifically using them in daily life - that's when you find out they really aren't used for any decisionmaking at all. They just like to see them.

 

[Mike Gibson]

IMHO, the more information from the SPC, the better. I want to hear what the experts think about severe wx events. In fact, I'd like the SPC to produce hourly high-resolution graphical hazard products (SWOHRn on the 5km grid) that fit in between the watch and warning timeframes (1-3hr). It could be generated from the HRRR or other high-res model.

People can choose whether they want to use or ignore the additional information.

Mike

 

[STurner]

I have seen several PDS tornado watches with a 70% probability of 2 or more tornadoes and a 40% chance of 1 or more (E)F2-F5 tornadoes. On the other hand I have seen several tornado watches not labeled as PDS's that had a 70% probability of 2 or more tornadoes and a 50% probability of 1 or more (E)F2-F5 tornadoes. Is this because of the higher probabilities of hail and wind on the tornado watch with a lower chance of (E)F2-F5 tornadoes than the one with a higher chance of (E)F2-F5 but has lower hail and wind probabilities. I am not sure why that would make a difference. Maybe it is SPC's criteria for one watch vs. another. Also it seems SPC changed the Public Information Statement as well. SPC used to issue one if there was a moderate risk of severe weather and the potential for strong tornadoes was 15% or more/ wind probailities were 45% or more. I have recently seen a couple Public Information Statements that were both high-end slight risk days that had a 30% or more chance of high winds and a 10% hatched areas for strong tornadoes. Is this also a new criteria the SPC has started doing as well for I dont remember any Public Information Statements that had probabilities of less than 45% or more chance of strong winds or 15% chance or more of strong tornadoes.

 

[Rich Thompson]

Shane,

There have been "PDS" tornado watches with 70/40 probs and "regular" tornado watches with 70/50 probs. The inconsistency originates within the SPC - the probabilities are assigned *after* the watch type decision is made, and we haven't established formal guidelines. We'll hopefully have more strict criteria for a PDS watch in effect soon (at least 60% for an EF2+ tornado).

The recent addition of the overnight PWOs for 10% SIG tornado areas came from discussions with the FL WFOs. There was a request to highlight the tornado threat when an isolated significant event might occur overnight when the threat for fatalities is greater, and the new PWOs were our response. A little blurb about it is still sitting on the main spc web page (under "other news").

 

[Rich Thompson]

Mike and Rob,

If I read your series of responses correctly, it's your belief that the presence of a tornado watch or PDS watch is good enough, and that gradations of probabilities within each watch type won't add anything useful. I can see your point, but I also don't see how the probabilities will be harmful. We didn't just start issuing watches at some new threshold, and you still have the same discussion accompanying the watch. If you reacted only when a watch was issued, then you keep doing the same thing in the era of the probabilities. This isn't about making sure everyone on the planet uses them, and it's not a response to some crazy external demand. We're simply sharing more of the information that went into the watch decision, without relying solely on a brief discussion that's written under serious time constraints.

 

[Mike Smith]

Mike and Rob,

If I read your series of responses correctly, it's your belief that the presence of a tornado watch or PDS watch is good enough, and that gradations of probabilities within each watch type won't add anything useful. I can see your point, but I also don't see how the probabilities will be harmful.

There are still a number of people (I have no scientific way of knowing how many) who believe the criteria for a PDS is one or more F4-5 tornadoes. I shared this with Joe a couple of years ago. When changes are made, people need to be educated on them and even I (and I watch this stuff pretty closely) didn't know a change had been made to the PDS criteria at the time it was made.

It will take decades to educate the public about this if you create more numbers and gradations. Uneducated users = greater chance of non-optimal decisions being made.

We didn't just start issuing watches at some new threshold, and you still have the same discussion accompanying the watch. If you reacted only when a watch was issued, then you keep doing the same thing in the era of the probabilities.

How do you know people won't be mislead? You may disagree, but I believe my comment about the "30%" probability of strong tornadoes in the Greensburg watch (see above) is a valid point. I think you are going to have a hard time calibrating people that 30% is a "high" probability which was what was issued immediately prior to one the most vicious tornado outbreaks in recent years.

This isn't about making sure everyone on the planet uses them, and it's not a response to some crazy external demand. We're simply sharing more of the information that went into the watch decision, without relying solely on a brief discussion that's written under serious time constraints.

Has SPC surveyed its users to see which they prefer, probabilities or a better discussion of the expected evolution of the threat? I would prefer an enhanced discussion, but I am a single user.

Rich, I really appreciate you have taken the time to engage us on these topics. Thank you.

 

[STurner]

Shane,

There have been "PDS" tornado watches with 70/40 probs and "regular" tornado watches with 70/50 probs. The inconsistency originates within the SPC - the probabilities are assigned *after* the watch type decision is made, and we haven't established formal guidelines. We'll hopefully have more strict criteria for a PDS watch in effect soon (at least 60% for an EF2+ tornado).

The recent addition of the overnight PWOs for 10% SIG tornado areas came from discussions with the FL WFOs. There was a request to highlight the tornado threat when an isolated significant event might occur overnight when the threat for fatalities is greater, and the new PWOs were our response. A little blurb about it is still sitting on the main spc web page (under "other news").

t
I just wanted to say thank you as well for you guys at SPC are trying to get the severe weather/tornado watches as accurate as possible. It just seems a little confusing at times.

 

[Greg Stumpf]

I'm way late on replying to this thread because I had to take a two-day break from ST. So these replies are aimed at some old posts, but I still feel the need to make them.

Greg, I posted information to several social science studies that indicate a lack of understanding of the PoP's. Did you miss it?

I didn't miss them, but they didn't answer my questions. But this response from you does:

With regard to how clients are able to understand/use them, it is because we work intensely with them to understand them and adapt them into their business processes.

Exactly - my argument being that the NWS currently does not do a good job of communicating uncertainty in forecasts, and thus the users don't have a good understanding. If your clients can understand, there is reason to believe that so can some public users.

They are motivated to understand, the public is not.

I'm sure many folks in that monolithic block you call "the public" would take great offense to this statement. Why do you feel absolutely certain there is zero hope?

they have the potential to do more harm than good. It is the latter point that concerns me the most.

This is why we need to find the best way to convey uncertainty in forecasts in order to elicit the appropriate response. Do you really think we are getting the correct response today? Why did those people die in Lone Grove OK? That storm was known to be a tornado threat well upstream from there, yet there is currently no official NWS product that could have been used to warn those folks that required greater than average lead time to take protection. Why not? Because the NWS currently feels that uncertainty in products with that great of lead time doesn't warrant an official product. I can think of other examples (e.g., Super Tuesday outbreak, Little Sioux, etc.) that motivate me to feel that the NWS could do much better in its dissemination of hazard information. Now imagine a future in which the NWS is producing longer lead time probabilistic hazard information on current threats. Then a private company could design a specific warning system for specific users requiring the longer lead time (high vulnerability) exploiting that hazard information, and the users of that system would never have to see the probability numbers!

I also feel (and this is definitely not necessarily the corporate opinion of my employer or my professional partners) that the methods that have been used to verify severe weather warnings have some serious flaws, and the NWS is not doing as good as they think they are. Just the fact that it is the NWS, and the issuing WFO, that is responsible for their own warning verification throws many red flags ("the fox guarding the henhouse").

Actually, am open to the concept of probabilities in certain circumstances, provided the probabilities are well-calibrated which is certainly (in my opinion) not the case now. So, they have the potential to do more harm than good.

The scientists agree with you 100% regarding calibration of probabilities. As Rich alluded to, the SPC does a lot of verification work to determine the accuracy of their probabilistic products - which were developed with a solid statistical foundation in collaboration with Dr. Harold Brooks (one can Google the references).

I have learned that the current program is amazingly effective, more than most meteorologists believe. So, I tend to be cautious about things that could damage that success.

I agree that the system has been very effective, and I'm also very sensitive about damaging the current system. But I also note that there have been some failures in the system recently as well as the potential for a massive failure if we have an major incident at locations where large numbers of people are gathered.

BTW - I am looking forward to reading your book when it becomes available.

I conclude that -- for the near to medium term -- let the private sector handle the probabilistic values for specialized users and be very cautious about tinkering with the public products.

I don't understand how you can be so opposed to probabilistic forecast information when you produce them yourself. Especially if you are concerned about calibration issues, information framing, etc.

Our concepts don't change the products, and we're very cognizant about keeping the legacy systems in place. We're simply adding more information about what the NWS forecasters are really thinking. And, as Rich said, it helps the forecasters to more robustly verify their forecasts.

 

[Robert Dewey]

I could see watch probabilities being of more value if they were represented graphically on a geographic map. For example, a watch may be several hundred miles long, with the north half having a greater probability for severe weather than the south half. Also, doesn't the size of the watch also affect probability, which would make the numbers appear skewed? For example, a watch that covers SE MI with a 40% TOR risk is actually indicating a higher probability - in terms of spatial distribution - than a watch that covers the entire state at 40% TOR. Do the forecasters take that into account? Do people that look at the products take that into account?

If Joe Blow sees a probability of 80% for the entire state of MI, he'll view that as significant... but what if that watch only covers half of the state with a probability of only 40%? Both watches actually indicate the same probability per square mile, but the end-user may perceive the threat for the 40% TOR as being less IMO.

 

[Bob Hartig]

I just tapped into this discussion and found it so engaging that I've read through the whole thread. Now I'd like to ask a question and connect it with a thought or two.

Who is the "public"?

As Greg has mentioned, that word takes in a group that is far from monolithic, so a one-size-fits-all summary of the public's awareness and its utilization of tornado percentages is, in my opinion, simplistic. I'm one of the public, and so are a good many other chasers who are neither professional meteorologists, nor EMT personnel, nor sociologists. We're educated on severe weather, and we make use of the various tools provided by the SPC; nevertheless, we're very much a subset of the public.

It was back in 2008 that I became aware of the tornado percentages, and since then I've consulted them on a conservative to moderate basis. I certainly appreciate their availability. The reason I haven't used them more is because, as Rich has pointed out, the link to them is buried amid various forecast discussions and is easy to forget about. As a result, I haven't trained myself to think of them as a frontline resource, the way I do with the SPC outlooks. It hasn't been a matter of usefulness for me, but of awareness, and I expect that's just the nature of experimental products. It's a matter of exposure.

Does storm chasing constitute a legitimate public use? I can't imagine why it wouldn't. On my part, when I look at the tornado percentages, they help to corroborate my own forecasts, or perhaps cause me to reexamine them. They don't chart my course for me, but they serve a purpose and I like that they're available. From my perspective, having more information is helpful, not harmful. Some chasers may use it, others may not, but there's nothing damaging about having a choice.

Would conducting a poll on Stormtrack be helpful?

 

[Rich Thompson]

I could see watch probabilities being of more value if they were represented graphically on a geographic map. For example, a watch may be several hundred miles long, with the north half having a greater probability for severe weather than the south half. Also, doesn't the size of the watch also affect probability, which would make the numbers appear skewed? For example, a watch that covers SE MI with a 40% TOR risk is actually indicating a higher probability - in terms of spatial distribution - than a watch that covers the entire state at 40% TOR. Do the forecasters take that into account? Do people that look at the products take that into account?

If Joe Blow sees a probability of 80% for the entire state of MI, he'll view that as significant... but what if that watch only covers half of the state with a probability of only 40%? Both watches actually indicate the same probability per square mile, but the end-user may perceive the threat for the 40% TOR as being less IMO.

The size of the watch does play a role, and that represents a weakness of our scheme right now, IMHO. There were proposals early on to issue some sort of updated "outlook style" probabilities within the watch area (grid), but the numbers were deemed too low for a watch. Thus, we went with the probs of the events in the entire watch to come up with larger numbers.

In practice, the only times the probs will vary greatly due to watch size will be with very large or very small watches.

I'll try to remember to check on the reliability and accuracy of the SIG tornado probs, per Rob's earlier question.

 

[Mike Smith]

One comment I'd like to make to perhaps close out my thoughts on this subject. The reason our clients are willing to learn the probability system is because they requested it and, for whatever reason (usually big dollars are involved) are motivated to learn a probabilistic system. I do not believe that is or will be the case with the general public. Most members of "the public" I come across view storm warnings as a necessary nuisance. They want to know, "do I go to the basement or not?" That is about the end of their interest.

So, I see neither the demand nor the interest in additional probability products for the public.

 

[Rich Thompson]

One comment I'd like to make to perhaps close out my thoughts on this subject. The reason our clients are willing to learn the probability system is because they requested it and, for whatever reason (usually big dollars are involved) are motivated to learn a probabilistic system. I do not believe that is or will be the case with the general public. Most members of "the public" I come across view storm warnings as a necessary nuisance. They want to know, "do I go to the basement or not?" That is about the end of their interest.

So, I see neither the demand nor the interest in additional probability products for the public.

I'll also take this opportunity to close out the discussion. The "public" consists of a large group of diverse interests. I agree that the vast majority of people off the streets don't know or care about watch probabilities, but they're not the target audience. SPC products are technical in nature and aimed at the meteorologists and weather enthusiasts that do care about meteorological details and forecast uncertainty. SPC provides the watch (and outlook) probability forecasts as additional information to weather-savvy folks that make decisions regarding severe thunderstorms.

As I mentioned in a previous post, SPC products only have a direct audience that is << 1% of the US population. The majority of people get their weather info through somewhere else (TV, radio, commercial web sites, etc.). If the watch probabilities can aid those who translate the forecasts into a format for public consumption, then we've done our job in helping to protect life and property. That's much different than worrying about how to educate the entire US on the scientific nuances of the specific SPC forecast products.

It's been an interesting exchange, and I appreciate the fact that it's remained entirely civilized

 

[Danny Neal]

Thanks guys, this conversation/thread had gone the exact way I had envisioned it going. After my first post and Rich's reply I had a couple more questions but held off in asking because I figured someone with more knowledge of the subject would word it better and they did. I was extremely pleased to get an inside look from a couple of the great minds working in the various weather forecasting and decision making capacities. Took a back seat and let the experts talk and at the end was not left unsatisfied. Thank you for your time all involved, probably one of the better ST threads we've had in a while. Goes to show what a little thought in ones reply can do! Thanks again!