Accuracy of watch probabilities

[Danny Neal]

Was reading the other discussion about how implementing %'s when predicting snowfall may be more user friendly to the public. I thought about what other products use %'s (other than the general 40% chance of rain) and thought about the SPC outlook/watch probabilities. This is not a general tool for the public, meaning they probably don't type in www.spc.noaa.gov to view these products, so my question is based towards those in the business. I was wondering the accuracies of these percents and would like to open up a discussion about it.

First, how does one determine the percents? In the figure I posted above, there is a 90% chance of 2 or more tornadoes. Which I figure is a 9 out of 10 chance that 2 or more tornadoes will occur. How do you figure that out? Gut feeling? Hail was 80% so an 8 out of 10 chance that 10 or more severe hail reports will be received. Is this based on model data, real-time meso-scale analysis, other? Not being critical in any way, just curious.

For each watch, probabilities for particular events inside the watch (listed above in each table) are determined by the issuing forecaster. The "Low" category contains probability values ranging from less than 2% to 20% (F2-F5 tornadoes), less than 5% to 20% (all other probabilities), "Moderate" from 30% to 60%, and "High" from 70% to greater than 95%. High values are bolded and lighter in color to provide awareness of an increased threat for a particular event.

After this is determined and the event has occurred, has this information been logged? Have accuracies been published? Would they be published under the accuracy of that forecaster or of the SPC in general?

I am kind of savvy about how the outlook probabilities work as its over a much larger area, but was wondering how they narrow down specifics....

Again, not trying to bash or condemn or be critical, just curious on the processes involved.
​

 

[rdale]

I asked SPC about that when I saw some strange percentages with southeast watches last year... They are all at the forecaster's preference. There's no real criteria used. If it is tracked - it's not on a publicly accessible site.

 

[Danny Neal]

I asked SPC about that when I saw some strange percentages with southeast watches last year... They are all at the forecaster's preference. There's no real criteria used. If it is tracked - it's not on a publicly accessible site.

I was afraid of that. I suppose keeping it internal would be better for all parties involved. (Well forecaster XXXXXX is better than forecaster ZZZZZZ type of deal). I guess my next question would be who utilizes these percentages? I am not with the media or directly connected to Emergency Management so I don't know what they decipher from SPC or NWS products/conference calls. I know my department, on a much smaller scale, gets all its' information indirectly from the NWS.

 

[Rich Thompson]

The SPC watch probabilities are different from the outlooks because the values apply to the entire watch, which can vary in size and is usually quite a bit larger than the grid size used in the convective outlooks. In effect, we treat the watch as a single grid box and assign odds to whether or not 2 or more tornadoes will occur during the time of the watch.

The numbers are assigned subjectively, which means the forecaster uses his/her best guess based on experience and understanding. Accordingly, not all forecasters will assign the same values for the same situation due to varying interpretations. I cannot quote values for the other forecasters, but I can say that the unit average is close to reliable for the tornado probabilities. That means when we say 90% chance of two or more tornadoes in the watch, two or more tornadoes are actually reported in the watch close to 90% of the time. The variations amongst forecasters is more apparent in the number of watches issued at various probabilities. We allow tornado watches at a lower threshold of 30% for 2+ tornadoes, while the default verification rate for all tornado watches over a multi-year period is 40% and 20% for 2+ and 1+ sig tornadoes, respectively. Those numbers are the first guess for each tornado watch, so you'll often see 40/20 watches when the forecaster thinks of it as a rather "typical" tornado watch situation.

In the not-to-distant future, we'll have the watch wording and the probabilities linked (see the (test) tab for http://www.spc.noaa.gov/products/watch/2009/ww0789.html ). The rules aren't official at the moment, but we'll eventually have the watch type and wording fixed for particular probabilities.

 

[Danny Neal]

Very good Rich, that is exactly what I wanted to know. Thank you for your input and good luck with 2010! I can't wait to see the new product.

 

[rdale]

I've never seen them utilized in media or emergency management other than the occasional "xx % chance of a major tornado with this setup?" kind of comments. I've never seen them change any operational decisionmaking.

 

[Tim Stoecklein]

I've heard the %'s quoted on numerous occasions by both TV and radio media in E CO/W KS, and the #'s do pop up in our daily wx briefings for local Skywarn during the spring or summer. The numbers are noticed...

 

[rdale]

the #'s do pop up in our daily wx briefings for local Skywarn during the spring or summer. The numbers are noticed...

The odds only come out when a watch is issued - are you sure that's what you are hearing in your daily briefings or are you talking about the convective outlook probabilities?

 

[Tim Stoecklein]

Yes, I'm talking about the probabilities issued with watches. No, they don't come up EVERY daily wx briefing, but our daily net takes place at 1830z so usually we're right on the fringe of when watches tend to get issued for our neck of the front range.

 

[rdale]

Gotcha. Out of curiosity - how do the different odds change your response / spotting / recovery planning / etc.?

- Rob

 

[Tim Stoecklein]

In general, I'd say the Watch probs are reacted to similar to POP's....10% chance of precip - I probably don't throw in my rain jacket....90% POP - I'll probably throw in my rain jacket, but leave the umbrella.

Basically I think the numbers feed into our group's level of awareness rather than directly impacting operational procedure....ie the bird dog getting "birdy" vs going on point. It's just to uncertain to predict a significant tornado (or #of TOR) to use the probabilities to influence EM setting up a staging area versus being aware...

 

[rdale]

So if the watch probabilities for a tornado are less than xx%, you don't deploy spotters into the field, and if it's higher you do? Who sets that number?

 

[Tim Stoecklein]

Nets are brought up, and mobile spotters deployed, at the request of served agencies (ie NWS). I'd imagine they're not using watch probability numbers to determine when to make their requests

Probability %'s are not used as thresholds for our local Skywarn group, I'm just saying I've heard/seen them used as an FYI type item...

 

[Mike Smith]

A Challenge

Some of you are aware of the discussion Greg Stumpf and I are having about probabilities in the Winter Weather Forum. Greg believes additional probabilities for additional products are of value and I believe they are superfluous.

Here is my challenge: Please specifically cite how the watch probabilities are used to make concrete decisions and the name of the agency and its location. For example, "we deploy spotters when the tornado threshold listed in the watch probability table exceeds 15%, below 15% we do not."

I would like to learn whether the probabilities are used for specific decision making or whether they are merely "interesting."

 

[Rich Thompson]

Here is my challenge: Please specifically cite how the watch probabilities are used to make concrete decisions and the name of the agency and its location. For example, "we deploy spotters when the tornado threshold listed in the watch probability table exceeds 15%, below 15% we do not."

I would like to learn whether the probabilities are used for specific decision making or whether they are merely "interesting."

Mike,

I assume you realize the watch probabilities are simply experimental at this time? Or, are we to expect someone to make decisions based on a product that is unfamiliar?

You don't like probabilities (they're "superfluous"), and that's your apparent preference. However, your preference does not change the fact that uncertainty is inherent in weather forecasts, and probabilities are the direct language of uncertainty. Why don't you pose your same question w.r.t. to PoPs?

On a related note, how would you propose that forecasters decide on categorical thresholds? Probabilities provide a direct and verifiable means of assessing uncertainty. As you approach perfection, probability values will trend toward zero and 100%. Are you against the expression of uncertainty?