Accuracy of watch probabilities

[rdale]

Now, try to envision a probability of tornado, probability of damaging winds (and straight winds of more than 100 mph were measured about the time of the radar image), and probability of large hail (and 2 3/4 inch hail, driven by 80+ mph winds were reported) all on a television screen

What TV met in their right mind would do such a thing? I'd use something more along the lines of the current pathcasting.

I think if that's your preferred method of displaying severe weather information, you're going to lose all your viewers to the other stations who have meteorologists who know what they're doing during severe weather...

 

[Mike Smith]

But, Rob, then you are back to the binary system.

Yes, you would do pathcasts on the complex of radar echoes I showed in the post above. But, the moment you add probabilities to the pathcasts the complexity skyrockets.

More to the point: Why do we need this? There is no doubt -- none -- that the vast majority of people, when in a tornado warning and being told by television meteorologists to "take cover" do just that. Multiple social science studies back up that contention.

Right now, we treat a tornado warning as a "100% event" -- take cover!! By introducing probabilities we introduce uncertainty as well as complexity. Chuck Doswell told me that, 15 minutes before the City of Greensburg was hit, he would have assigned a 70% chance (in a probabilistic warning) to Greensburg.

So, in the binary system we are telling Greensburg TAKE COVER (100%). Chuck said he would have told them 70%. Please explain to me how people are more likely to take cover and save their lives with 70% as opposed to 100%??

With the current system, as far as I can tell, 99+% of the city was in shelter. Would that have been the result if we had told them, 15 minutes before, the probability was 70%?

 

[rdale]

But, Rob, then you are back to the binary system.

For Joe Public -- yes. For Joe Ripnread weathercaster, it's a lot easier to train him on interpreting probabilities than it is to interpret radar. For Joe EMA, it's a lot easier for him to send the spotters to the 70% threat zone than it is to give him a polygon and say "put them in there." For Joe School Superintendent, he can see that there's a storm 70 miles away at 1pm with a 25% chance of being in his town at 3pm, so he can delay putting them on the bus. Right now he either is the type that says "tornado warning 100 miles to our west, let's send them home now" or the type that doesn't do anything until the weather radio alerts.

There is no doubt -- none -- that the vast majority of people, when in a tornado warning and being told by television meteorologists to "take cover" do just that. Multiple social science studies back up that contention.

I agree 100%. I don't intend the general public to ever see these probabilities. But I do think specialized fields in the response sector (and computers) can take great advantage of this data, which always existed but was never able to be put into computerized format.

 

[Mike Johnston]

Also, if you like the new "storm-based" warning outlines, then you like the probability approach. Why does the warning outline widen as you get farther away from the initial position? Because the probability of the tornado being in a particular location farther out in time decreases (the forecast position is uncertain).


Mike

No, I do like the storm-based warning outlines, but that doesn't necessarily mean I have to like the probability approach. Prior to the polygons, the warnings were issued on a county basis. Political boundaries being irrelevant to the behavior of the atmosphere, I appreciate the new polygon warnings were an improvement.

Now, I appreciate there is some implied probability in any forecast. All I'm questioning is: at what point does a system that requires the forecaster to make his probability explicit really provide sufficient benefit to justify the cost? The "cost", to me, is diversion of time, effort, and attention in a short-fuse situation.

 

[Bobby Prentice]

Sorry I'm late in responding to this thread but I see Rich Thompson and Greg Stumpf have been doing a fine job of taking up the cause of probabilistic forecasting in my stead. Here's my input:

NWS customers constitute a wide spectrum (public, private, government, military, etc.) and it's impossible to create a "one-size fits all" forecast product. The NWS would do a disservice to its customers (taxpayers) if it didn't forecast to the absolute best of its ability. Probabilistic forecasting enables the NWS to do its best job. It allows for many different versions of the same forecast, tailored to the end-user's special needs.

Probabilistic forecasts are a potential boon to private weather companies too. They can be the ones to take the highly-detailed probabilistic forecasts, create tailor-made products for specialized users, and then sell them to customers.

Just because most people do not yet fully understand (or care about) probabilistic forecasting does not minimize its potential benefits, particularly for sophisticated users. I suspect many/most storm chasers on ST already use probabilistic forecasts in their chase decisions. I know I do. I generally won't chase unless the SPC Probabilistic Outlook forecasts a 10% or greater chance of 2" hail (blue hatched hail area) which I equate with supercell storm modes. This sure beats trying to decipher the old-fashioned categorical SLGT RISK!

I'm proud to work with colleagues such as Rich and Greg who have the vision to see the benefits of and campaign for a probabilistic forecasting world. The best forecast is never black (yes) and white (no), but rather shades of gray (probabilities).

Chuck Doswell has a great essay on this subject: The Unbearable Burden of Certainty: An Essay in Favor of Subjective Probabilities in Weather Forecasting by Chuck Doswell

 

[Mike Smith]

The best forecast is never black (yes) and white (no), but rather shades of gray (probabilities).

Virtually no one outside of the meteorological profession would agree with that statement, which is the disconnect between the probability advocates and those of us who believe the effort should be put into improving accuracy.

The goal (which may never be attainable) should be 100% of the people who have their homes (or whatever) destroyed by a tornado be in a warning and there should be zero, or very few, false alarms.

The public (as exemplified by yesterday's Kiwanis audience) views rain PoPs as a "cop out." It will be even worse with tornado PoPs.

 

[rdale]

he disconnect between the probability advocates and those of us who believe the effort should be put into improving accuracy.

Why are those mutually exclusive? Wouldn't it make sense that if you make your probabilities better, you are making your accuracy better too?

The public (as exemplified by yesterday's Kiwanis audience) views rain PoPs as a "cop out." It will be even worse with tornado PoPs.

Since the general public won't see the tornado "PoPs" -- how would it be worse? Wouldn't it be better since the people responsible for protecting the public would have more info?

Again let's go back to the beginning. If all you want is the "YES or NO" tornado warning, it will still be there for the public. If you want/need more than that, you can have that too. I fail to see the negative?

 

[Mike Smith]

In spite of the disclaimers in this thread, there are those who plan to make probabilistic storm warnings for the public-at-large. Look at the document linked to in Post #57 in this thread. It states...

A typical year brings 6 hurricanes, 1200 tornadoes, 5000 floods, 10,000 violent thunderstorms, and various other hydrometeorological and related threats (e.g. wild fires) to the United States causing on average 500 deaths, 5000 injuries, and approximately $14 billion in losses each year. Shifting to probabilistic forecasts and a hazardous weather and water warning capability which incorporates probabilistic forecasts and thresholds into the warning criteria, a “warn on forecast” (or “warn on probability”) capability, could increase warning lead times and provide emergency managers, other decision makers and the public other valuable information by which to save lives and property.

The official position of this AMS document is that we should transition to a probabilistic warning system for the public.

If I were confident that we will not convert to probability warnings for the public, I would not be spending all of this time on this discussion thread. It is not just the AMS. Unless I misheard, John Snow kicked off the December, 2008, warning workshop by saying we were going to be doing probability warnings for the public in five years.

While I believe there is less value in probability-based warning messages for EM's (see my Friday evening Greensburg example above) and other "special" users than the probability advocates participating in this discussion, I concede there could be some value there that I am not seeing.

That said, I believe probability-based warnings for the public (at 9:40pm Friday, May 4, 2007: "70% chance of a tornado in Greensburg") is a dangerous way to try to eliminate the remaining 2% of tornado fatalities (when compared with the death rate before the watch/warning system was developed). By going forward with this proposed overhaul of the public warning system, we risk confusing people -- thus putting at risk -- the 98% of deaths we are now preventing.

 

[rdale]

I don't see where the AMS is calling for anyone to drop the current public warning system and deliver it to them exclusively in probabilities, although I admit I haven't read the entire document. From what I read, they want to enhance the current system by expressing probabilities where appropriate.

On the other hand - the AMS is no longer connected to actual forecasters on the front lines. They are a research organization, so regardless of what the AMS says (paging Dr. Cullen) it won't be done solely on their call.

Five years ago people said that polygon warnings would never work. Too confusing. The system already is as good as it can get.

Oops

Using polygons you can still get the plain-old county warning if you want. But IF YOU WANT MORE INFORMATION, and aren't trained in radar interp, you have the polygon.

 

[Bobby Prentice]

While I believe there is less value in probability-based warning messages for EM's (see my Friday evening Greensburg example above) and other "special" users than the probability advocates participating in this discussion, I concede there could be some value there that I am not seeing.

I see lots of benefits of probability-based warning forecasts for EM's, corporations, government, military, and other "specialty users." A limitless number of products that enterprising weather salesman could sell to many new potential customers.

How about this severe weather scenario:

A supercell thunderstorm forms 2 hours (T-120 minutes) southwest of of Wichita and the NWS issues a probabilistic tornado warning graphic for its path that indicates a 1 percent chance of a tornado hitting within 5 miles of the center of McConnell AFB. At T-90 minutes the warning is updated to a 2% chance. At T-60 minutes, a 5% chance. At T-30 minutes, a 10%. At T-5 minutes, a 99% chance.

At what point should the commanders scramble the billions of dollars worth of aircraft sitting on the tarmac and sound the sirens at the base? What is the NWS's probability threshold for issuing tornado warnings in the current format? Who the heck knows!? Let's say it was 20%. In the above scenario, by the time McConnell AFB met NWS tornado warning criteria (T-5 minutes), it's too late to scramble the aircraft, they get destroyed, and the military decides to shut the base down.

This scenario isn't far-fetched. It happened on April 26, 1991 when the F-5 Andover tornado struck the base. Fortunately, the B-1 bombers were missed by a few hundred yards and the base stayed open, saving Wichita from an economic disaster.

Perhaps the base commanders could have had standing orders that at T-90 minutes (the time needed to scramble all aircraft) if the tornado probability threshold is 1% or higher, the aircraft would be scrambled out. They certainly wouldn't have gotten this detailed information from the current tornado warning format.

Notice, I haven't even mentioned the corresponding hail probability forecast which can cause extreme damage to aircraft as well, but is basically overlooked in the current NWS tornado warning format. Notice, I could have exchanged "McConnell AFB" with "large public venue" and "aircraft" with "people."

Let's not restrict our probability forecasts to just severe weather:

- How about a 24-hour 1-inch rainfall probability forecast for the basin that feeds a hydroelectric power plant? The plant uses the data to decide how much power to generate on that day. A former co-worker of mine was basically making these types of multi-million dollar decisions for the Tennessee Valley Authority (TVA) many years ago.

- How an hour-by-hour maximum icing potential probability forecast at various altitudes for aviation? If the value exceeds 50%, aircraft will avoid that airspace.

- Or a 6-hour 10 knot+ mean wind probability forecast for a field of wind power-generation turbines? If the probability is less than 30%, the utility will replace the blades instead.

- Or a 500 W/m*2 daily solar radiation probability forecast for a solar panel power-plant? If the value is less than 50% the company will do maintenance on that day instead.

- Or a 3-hour 15 knot wind probability forecast for a local sail boat club. If the probability is less than 70%, they won't waste their time.

The possible products and uses are limitless.

 

[Jonathan Garner]

Mr. Smith continues to use Greensburg as an example in his argument/defense against probabilistic forecasting of tornadoes during warning mode...stating that a 70% probability would in affect mis-respresent the true threat associated with that warning. However, if you consider the fact that if a forecaster were to assign a 70% probability of a tornado affecting any particular square mile of the nation, then from a climatological perspective, there is a fairly high degree of certainty that the tornado threat is quite large (the threat is many times above climatology). It would then be up to sophisticated media outlets to relay the heightened threat to the public (i.e., those media who know what kind of information the warning forecaster is indicating with that probability). A 70% probability assigned to a tornado warning would be very significant, while on the other hand, if the forecaster has a large degree of uncertainty whether a tornado has even formed (based on marginal radar signatures, for example) then a lower probability would be called for. Probability forecasting is the natural way to convey this certainty, and can even be assigned within a cone, in order to indicate the highest spatial threat as well. There is so much more value in a probabilitistic forecast versus a simple yes/no forecast.

 

[Mike Smith]

Some Background and a Suggestion

Bobby,

You, Rich and Greg write as if these products are not being created and delivered now. They are by private sector weather companies. The fact the NWS doesn't do something, doesn't mean it isn't done. I really believe (and I mean this in a constructive, encouraging manner) that you guys need to get out of Norman more and see what the rest of the field is doing. You'd be amazed what we do and how advanced we are! I'll make you and any of your Norman NWS/OU/NSSL colleagues an offer: Come up February 11 for our AMS luncheon meeting and I'll give you the grand tour of WeatherData after. The meeting location, Sedgwick Co. Emergency Management, and WeatherData are just a few blocks from each other.

I am very familiar with the Wichita-Andover close call with regard to the B-1's and KC-135's at IAB. With all due respect, I believe you have the approach backward: No base commander is going to scramble the aircraft with a 5% probability and, by the time it goes above 10%, it is too late (given your scenario). I quote the late Col. Bob Miller in my book on this topic. Scrambling expensive aircraft, given a 5% chance, would lead to the base commander being "laughed out of this man's Air Force." When you visit WeatherData, as I hope you will, I'll fill you in on some background information on this topic.

I have never been against all uses of probabilities as a reading of this entire thread will reveal. I am against upending the current watch/warning system for the public. Telling people there is a 10% chance of a tornado in 30 minutes is a great way to convince them not to go to shelter. It is amazing to me that so many in Norman (and I am aware of others) want to fix a system that is most certainly is not broken.

At the request of the ICT AMS members, I am doing Miracle at Greensburg for that meeting. The Greensburg presentation has been viewed by the mayor of Greensburg, the head of the Kiowa County Commission, and 200+ citizens of Greensburg. It has been very well received and I have been praised for its accuracy. I sincerely believe you will be even prouder to be part of the meteorological profession once you see it.

The time you would spend to see the presentation and tour WeatherData would probably be time well-spent. Call me at (316) 265-9127 or email me at msmith At weatherdata Dot Com if you would like to set it up. We'd be glad to have you!!

Respectfully,
Mike

 

[Mike Gibson]

You, Rich and Greg write as if these products are not being created and delivered now. They are by private sector weather companies. The fact the NWS doesn't do something, doesn't mean it isn't done. I really believe (and I mean this in a constructive, encouraging manner) that you guys need to get out of Norman more and see what the rest of the field is doing. You'd be amazed what we do and how advanced we are!

That logic would have NOAA discontinue the NEXRADs, models, etc. because all that can be done by private companies, too.

I have never been against all uses of probabilities as a reading of this entire thread will reveal. I am against upending the current watch/warning system for the public.

I haven't seen anything that suggests the NWS would upend/discontinue the current public watch/warning implementation. Rather, this would be a way to distribute additional raw data.

I guess I fail to see what gets some people so up in arms over distributing probabilities. Were people similarly upset when the HRW-NMM/ARW 4km forecast reflectivity graphics were made available on NOAA websites??? They would seem to be worse for "end users" since people could look at them and think that they wouldn't be hit by a storm at all, e.g. "see, this big ol' storm goes 5 miles to my south at 3hrs and there's nothing over the lake, let's jump in my boat and do some fishing!".

Mike

 

[Mike Smith]

That logic would have NOAA discontinue the NEXRADs, models, etc. because all that can be done by private companies, too.

Mike,

I don't see this, at all, as a public vs. private discussion. I was responding to Bobby's list of potential products. At least some of them (perhaps all) are already being provided by private sector companies. And, since he was referring to specific industries (i.e., wind generation) those are, under current policy, the purview of the private sector. But, again, to me this is not a public/private discussion.

I haven't seen anything that suggests the NWS would upend/discontinue the current public watch/warning implementation. Rather, this would be a way to distribute additional raw data.

Mike, read the section of the AMS report I quoted: "Shifting" to probabilistic warnings. That is their word, not mine.

The "shifting" phraseology was also used in the breakout groups at the warnings conference in Norman in December, 2008. What was really interesting about that is there were two TV mets, two private sector mets, two EM's and two social scientists and every member of that group was against probability warnings. Each member of the Norman subgroup was in favor.

I received an unsolicited email this morning from a person who has been following this thread but didn't want to post. The person who sent the email wishes not to reveal his identity to the storm track group but he is an EM in the tornado belt. He says,

wanted to pass my total agreement with you along on the NWSs attempted push of more probability based products to the public. The line: "I really believe (and I mean this in a constructive, encouraging manner) that you guys need to get out of Norman more and see what the rest of the field is doing." is exactly right...

We have people that don't understand the warning process as it is, let alone when you install a whole new layer of unprecedented complexity and forecaster objectivity into the process. To the mathheads that have never left Norman it might add more value, but to Joe Public it's not going to improve anything!

Just wanted to pass along my support of your position!

We have a system that has been amazingly successful and saves 98% of the potential deaths from tornadoes. Why do we want to "fix" a system that is obviously not broken?!

Mike

 

[rdale]

It looks like he read the thread wrong - nobody in NWS (at least in this thread) wants to replace the current warning system with probabilities. I'm not sure why that keeps coming back up.

Nobody is going to change the warning system in ANY fashion without plenty of discussions with the EMs / TV mets / etc. But again, I fail to see where anyone has said the current warning system will be REPLACED by probabilities. Only enhanced.

When the AMS creates policy for NWS HQ, then I guess we have an issue. But I've not heard of any push for the AMS President to knock off the NWS administration, so I don't believe their policy statements are related to the future of meteorology.