Forecasting Accuracy 2024 - Good or Bad?

[Jason N]

I guess some of that will come down to how many Sig Tors there were yesterday, I would say 1 for sure, perhaps 2? but I stopped watching after the Barnell storm, so I don't know if anything formed after that. if it ends up being just one report of a sig tor, is that considered enough to call it a hit? .. I mean, their reasoning for upgrading to, valid. But I wonder when they updated it last evening to trim the area, should they have just gotten rid of the High and left MDT. but here I am, hind sighting lol.

One thing they don't do is discriminate with SLIGHT, MDT, or HIGH on whether its tornadic or not, their categories are irrespective of tornadoes, just severe, and longevity. So based on those conditions, they probably hit what they forecast.

 

[Jesse Risley]

I guess some of that will come down to how many Sig Tors there were yesterday, I would say 1 for sure, perhaps 2? but I stopped watching after the Barnell storm, so I don't know if anything formed after that. if it ends up being just one report of a sig tor, is that considered enough to call it a hit? .. I mean, their reasoning for upgrading to, valid. But I wonder when they updated it last evening to trim the area, should they have just gotten rid of the High and left MDT. but here I am, hind sighting lol.

One thing they don't do is discriminate with SLIGHT, MDT, or HIGH on whether its tornadic or not, their categories are irrespective of tornadoes, just severe, and longevity. So based on those conditions, they probably hit what they forecast.

I haven't seen a good, purely objective categorization of what consists of a "bust" per se. For the high risk, it may come down to how the wind damage reports are quantified (see below). If we don't see "numerous intense and long-tracked tornadoes" once all of the cards are on the table, then the specific, tornado-driven high risk upgrade wouldn't verify. I have no idea how widespread and intense the wind damage is in eastern OK though. I wouldn't personally call the event a "bust," since there were a plethora of severe weather reports, although it's possible that the final report verifications will merit that the event only met a lower categorical threshold for verification, e.g., an enhanced or moderate, for example. I think it's important to message that this doesn't mean a complete bust even though a specific severe weather outlook categorization threshold may not have been met, although I understand the social science optics with the general public are far more complex.

• 5-HIGH (magenta) - High risk - An area where a severe weather outbreak is expected from either numerous intense and long-tracked tornadoes or a long-lived derecho-producing thunderstorm complex that produces hurricane-force wind gusts and widespread damage. This risk is reserved for when high confidence exists in widespread coverage of severe weather with embedded instances of extreme severe (i.e., violent tornadoes or very damaging convective wind events).

 

[Jason N]

I think it's important to message that this doesn't mean a complete bust even though a specific severe weather outlook categorization threshold may not have been met, although I understand the social science optics with the general public are far more complex.

I dont think it was a complete bust at all. but when it comes to whether someone has to calculate a HIT/MISS/FAR, it would be interesting to see how they determine it, separate from the social aspect ofcourse, as you said thats too complex and I would agree with you, better there to just take a pole before and after storm outbreaks maybe on , did you feel you were adequately warned? type questions. lol

 

[Jesse Risley]

@Jeff Duda posted this in another thread. I was previously unaware of Victor's page:
https://atlas.niu.edu/pperfect/

 

[Jason N]

I cant get to the page. is there another link? and is there any information on what's being used to produce this?

 

[Jesse Risley]

I cant get to the page. is there another link? and is there any information on what's being used to produce this?

https://www.atmos.albany.edu/facstaff/rfovell/NWP/bryan-fritsch-2000.pdf

 

[Jason N]

you sent me the link to MAUL lol , I was talking about the practically perfect Probs

 

[Jesse Risley]

you sent me the link to MAUL lol , I was talking about the practically perfect Probs

Oh, oops: SPC Practically Perfect Hindcast

 

[Jason N]

hahaha , no worries.. thank ya!

 

[Sean Ramsey]

Kind of funny how we treat the weather risk communication after the fact as opposed to say fire danger risk or other areas where danger can occur.

It's a risk, doesn't mean when it doesn't happen it's wrong. They assign high fire danger risk in forests all the time, no one goes back and says bust if a fire didn't occur on a particular day. Climbing Mount Everest is a high risk, but if climbers didn't die on a particular day, no one says the risk didn't verify.

I understand that people modify their days based on weather forecast communication. But, there was an inherent danger yesterday, it just wasn't realized on a mass scale.

Is the forecast made to predict tornadoes (quantity, size & type) or to communicate the risk (or danger) associated with them if they occur, or both?

Seems to me, the (very good) chance was there, right or wrong after the fact. Maybe someday they'll reach a point in forecasting where it's called "High Probability".

 

[Jason N]

It's a risk, doesn't mean when it doesn't happen it's wrong. They assign high fire danger risk in forests all the time, no one goes back and says bust if a fire didn't occur on a particular day.

Yeah, I think that's where we get into the social aspect of the discussion, Warning for Warnings sake, be damned the statistics. I can see that on several levels, I just know that there are others within communities, that look at justification, dollars, Insurance, business, Operations, So say yesterday was only a Marginal Risk, people died and towns were leveled. They might ask, "well, you missed a 2 billion dollar severe weather event", why?

I think in that instance, people might not get away with, "well it's just risk", they might start asking more of how your performance as a center of excellence is, why are you here?, because you cost the taxpayer money to exist. Beyond it being important to potentially save lives, there is billions in industry that rely on it as well. So, I think both sides of the discussion have a say on some level.

But Human to Human, I fall on the side that says, it's all informative! treat it as such. we shouldn't care too much about how they performed, but believe me, someone does.

 

[Jesse Risley]

Though none of my own original thoughts per se, I always try to stress that meteorologists are expected to predict the future of how a fluid dynamic will behave. It's as simple as that. It's not an exact science and probably never will be. If the general public understood the nature of fluid dynamics (most get the concept of fortune telling, although that's a completely false science even at best) and the probabilities involved with trying to predict future behaviors, it would probably be better understood, but we'll probably never buck the whole "it must be nice to get paid to be wrong half the time" poppycock.

 

[Jeff Duda]

Yesterday's High risk forecast was a bust. I don't see how you can't justify that. Now, I don't know how some differentiate between a "complete bust" and a "regular bust", so maybe the nuance saves that discussion. But a high risk event is supposed to be a rare event that only occurs a few times per year and consists of widespread and extreme severe weather reports. "Widespread" and "extreme" are subjective terms, sure, but even looking at the raw LSR counts shows that what happened yesterday has already been exceeded in 2024 by non-high risk events such as March 14th (Storm Prediction Center 20240314's Storm Reports). Considering even further that the official "high" designation was due strictly to the tornado probabilities and not hail or wind, and the forecast was errant even more so since it overforecast one category and underforecast another.

I've noticed that when SPC issues PDS tornado watches they tend to be very large (looking into finding stats for this). But making a watch large in area artificially increases the chances of it capturing >=2 tornadoes and >=1 EF2+ tornadoes. [Side note: 2/1 tor/sigtor already seems like a pretty low bar, and many non-PDS tornado watches readily exceed that threshold, so I'm already not sure the PDS designation is that helpful.] I would think, if anything, you'd want to issue smaller PDS tornado watches to really focus in on the highest threat areas without overly alerting way more people. IDK, I am not an SPC forecaster and cannot speak for their forecasting philosophy, but in this regard I disagree with it.

Secondly, this particular PDS tornado watch (Storm Prediction Center PDS Tornado Watch 189) is likely only going to verify by the skin of its teeth, seeing as the only likely sig tor at this point occurred in the far eastern end of Osage County, and even the parallelogram defining the watch seems to end right about at the longitude of where Barnsdall is. So even with that extra huge area, they still almost missed! That to me does not show a particularly high level of skill.

 

[Jason N]

Yesterday's High risk forecast was a bust. I don't see how you can't justify that.

This is what I mean:

is it a bust in skill/accuracy of the watch polygon? ehem, parallelogram...... or, is it a bust in reports to spatial coverage of the high area? or to the specifics of you didnt get but 1 EF-3 or higher! you suck!! lol.... you mention how it was superseded in count on a previous forecast, so, it seems to me like, we look to verify forecasts, we do that now with reports so if we were to somehow link that, say..... (Marginal = 5-9 reports ; Slight 10-19 reports; Enhanced 20-29 reports; MDT 30-49 reports; High 50+) its arbitrary just for discussion, and the reports are aggregate totals inside of the spatial area of coverage for each, not separated into each category, it seems like this would be the easiest way to see if an area Hit or Missed

 

[Jesse Risley]

This is what I mean:

is it a bust in skill/accuracy of the watch polygon? ehem, parallelogram...... or, is it a bust in reports to spatial coverage of the high area? or to the specifics of you didnt get but 1 EF-3 or higher! you suck!! lol.... you mention how it was superseded in count on a previous forecast, so, it seems to me like, we look to verify forecasts, we do that now with reports so if we were to somehow link that, say..... (Marginal = 5-9 reports ; Slight 10-19 reports; Enhanced 20-29 reports; MDT 30-49 reports; High 50+) its arbitrary just for discussion, and the reports are aggregate totals inside of the spatial area of coverage for each, not separated into each category, it seems like this would be the easiest way to see if an area Hit or Missed

I'd argue that it's two-pronged: it was largely a tornado-driven high risk and those metrics were not met based on the current (and projected) reports and, spatially, the wind reports that would have verified a high risk for damaging wind events were displaced from the delineated high risk area. Those two components fell short of what was forecast. There were plenty of severe weather reports, so it's not like a true bust where severe weather fails to completely materialize as forecast, but that gets into the nuances of language and subjectivity.