Jesse Risley
Staff member
Forecasting Accuracy 2024 - Good or Bad?
- Thread starter Warren Faidley
- Start date
Jason N
EF4
you sent me the link to MAUL lol , I was talking about the practically perfect Probs
Jesse Risley
Staff member
Jason N
EF4
hahaha , no worries.. thank ya!
Sean Ramsey
EF4
Kind of funny how we treat the weather risk communication after the fact as opposed to say fire danger risk or other areas where danger can occur.
It's a risk, doesn't mean when it doesn't happen it's wrong. They assign high fire danger risk in forests all the time, no one goes back and says bust if a fire didn't occur on a particular day. Climbing Mount Everest is a high risk, but if climbers didn't die on a particular day, no one says the risk didn't verify.
I understand that people modify their days based on weather forecast communication. But, there was an inherent danger yesterday, it just wasn't realized on a mass scale.
Is the forecast made to predict tornadoes (quantity, size & type) or to communicate the risk (or danger) associated with them if they occur, or both?
Seems to me, the (very good) chance was there, right or wrong after the fact. Maybe someday they'll reach a point in forecasting where it's called "High Probability".
It's a risk, doesn't mean when it doesn't happen it's wrong. They assign high fire danger risk in forests all the time, no one goes back and says bust if a fire didn't occur on a particular day. Climbing Mount Everest is a high risk, but if climbers didn't die on a particular day, no one says the risk didn't verify.
I understand that people modify their days based on weather forecast communication. But, there was an inherent danger yesterday, it just wasn't realized on a mass scale.
Is the forecast made to predict tornadoes (quantity, size & type) or to communicate the risk (or danger) associated with them if they occur, or both?
Seems to me, the (very good) chance was there, right or wrong after the fact. Maybe someday they'll reach a point in forecasting where it's called "High Probability".
Jason N
EF4
Yeah, I think that's where we get into the social aspect of the discussion, Warning for Warnings sake, be damned the statistics. I can see that on several levels, I just know that there are others within communities, that look at justification, dollars, Insurance, business, Operations, So say yesterday was only a Marginal Risk, people died and towns were leveled. They might ask, "well, you missed a 2 billion dollar severe weather event", why?
I think in that instance, people might not get away with, "well it's just risk", they might start asking more of how your performance as a center of excellence is, why are you here?, because you cost the taxpayer money to exist. Beyond it being important to potentially save lives, there is billions in industry that rely on it as well. So, I think both sides of the discussion have a say on some level.
But Human to Human, I fall on the side that says, it's all informative! treat it as such. we shouldn't care too much about how they performed, but believe me, someone does.
Jesse Risley
Staff member
Though none of my own original thoughts per se, I always try to stress that meteorologists are expected to predict the future of how a fluid dynamic will behave. It's as simple as that. It's not an exact science and probably never will be. If the general public understood the nature of fluid dynamics (most get the concept of fortune telling, although that's a completely false science even at best) and the probabilities involved with trying to predict future behaviors, it would probably be better understood, but we'll probably never buck the whole "it must be nice to get paid to be wrong half the time" poppycock.
Yesterday's High risk forecast was a bust. I don't see how you can't justify that. Now, I don't know how some differentiate between a "complete bust" and a "regular bust", so maybe the nuance saves that discussion. But a high risk event is supposed to be a rare event that only occurs a few times per year and consists of widespread and extreme severe weather reports. "Widespread" and "extreme" are subjective terms, sure, but even looking at the raw LSR counts shows that what happened yesterday has already been exceeded in 2024 by non-high risk events such as March 14th (Storm Prediction Center 20240314's Storm Reports). Considering even further that the official "high" designation was due strictly to the tornado probabilities and not hail or wind, and the forecast was errant even more so since it overforecast one category and underforecast another.
I've noticed that when SPC issues PDS tornado watches they tend to be very large (looking into finding stats for this). But making a watch large in area artificially increases the chances of it capturing >=2 tornadoes and >=1 EF2+ tornadoes. [Side note: 2/1 tor/sigtor already seems like a pretty low bar, and many non-PDS tornado watches readily exceed that threshold, so I'm already not sure the PDS designation is that helpful.] I would think, if anything, you'd want to issue smaller PDS tornado watches to really focus in on the highest threat areas without overly alerting way more people. IDK, I am not an SPC forecaster and cannot speak for their forecasting philosophy, but in this regard I disagree with it.
Secondly, this particular PDS tornado watch (Storm Prediction Center PDS Tornado Watch 189) is likely only going to verify by the skin of its teeth, seeing as the only likely sig tor at this point occurred in the far eastern end of Osage County, and even the parallelogram defining the watch seems to end right about at the longitude of where Barnsdall is. So even with that extra huge area, they still almost missed! That to me does not show a particularly high level of skill.
I've noticed that when SPC issues PDS tornado watches they tend to be very large (looking into finding stats for this). But making a watch large in area artificially increases the chances of it capturing >=2 tornadoes and >=1 EF2+ tornadoes. [Side note: 2/1 tor/sigtor already seems like a pretty low bar, and many non-PDS tornado watches readily exceed that threshold, so I'm already not sure the PDS designation is that helpful.] I would think, if anything, you'd want to issue smaller PDS tornado watches to really focus in on the highest threat areas without overly alerting way more people. IDK, I am not an SPC forecaster and cannot speak for their forecasting philosophy, but in this regard I disagree with it.
Secondly, this particular PDS tornado watch (Storm Prediction Center PDS Tornado Watch 189) is likely only going to verify by the skin of its teeth, seeing as the only likely sig tor at this point occurred in the far eastern end of Osage County, and even the parallelogram defining the watch seems to end right about at the longitude of where Barnsdall is. So even with that extra huge area, they still almost missed! That to me does not show a particularly high level of skill.
Jason N
EF4
This is what I mean:
is it a bust in skill/accuracy of the watch polygon? ehem, parallelogram......
or, is it a bust in reports to spatial coverage of the high area? or to the specifics of you didnt get but 1 EF-3 or higher! you suck!! lol.... you mention how it was superseded in count on a previous forecast, so, it seems to me like, we look to verify forecasts, we do that now with reports so if we were to somehow link that, say..... (Marginal = 5-9 reports ; Slight 10-19 reports; Enhanced 20-29 reports; MDT 30-49 reports; High 50+) its arbitrary just for discussion, and the reports are aggregate totals inside of the spatial area of coverage for each, not separated into each category, it seems like this would be the easiest way to see if an area Hit or Missed
Jesse Risley
Staff member
This is what I mean:
is it a bust in skill/accuracy of the watch polygon? ehem, parallelogram......
I'd argue that it's two-pronged: it was largely a tornado-driven high risk and those metrics were not met based on the current (and projected) reports and, spatially, the wind reports that would have verified a high risk for damaging wind events were displaced from the delineated high risk area. Those two components fell short of what was forecast. There were plenty of severe weather reports, so it's not like a true bust where severe weather fails to completely materialize as forecast, but that gets into the nuances of language and subjectivity.
Jason N
EF4
all good points!..
Jamie H
EF2
Considering I thought it wasn't a bad initial forecast, the past few replies have been really useful, thanks!
JamesCaruso
Staff member
Some good points generally from a social science perspective and how people interpret risk - which is generally poorly. People focus only on the probability of something happening, forgetting that, unless the forecast probability is 100%, the forecast is implicitly assigning a non-zero probability to that thing NOT happening. I like the fire danger analogy.
But with regard to your last point about “high probability” - that’s already exactly what it’s supposed to be, a probabilistic forecast.
William Monfredo
EF2
Chased OK high-risks since 2010; that year & 2011 produced. Since then, as you know, not so much. (SPC's Roger Edwards said a degree too warm aloft May 20, 2019, Magnum day, prevented a major outbreak.) May 6, I don't think the storms interfered with each other, at least early on, but the surface temps seemed low for much of the day due to clouds. Yet, elevated instability can still get a job done; go figure. (Fall tornadoes in northern AZ anyone?)
Yesterday's event appears to have been slightly underforecast* by SPC given the practically perfect observation contour levels exceeded 60% for both wind and hail depending on the source (see below). According to the convective outlook probability-to-category conversion table, however, it appears that the dearth of significant wind reports would have rendered this forecast failing to reach the high risk category (by the narrowest of margins). And apparently there is no PP contour level for hail LSRs that is high enough to verify a high risk. *Since moderate risks were in place based on both wind and hail, technically SPC's day 1 forecast yesterday was pretty accurate. But in my biased opinion it still seems a bit low based on the coverage of the 45 and 60% contours, both in terms of spatial extent and location. Also, neither hazard type was forecast to exceed 60%.
Practically perfect obs:
Scaling is a bitch:
There are 421 filtered LSRs as of 10:45 AM MDT the day after this event.
Meanwhile, Monday's event still only has 355 filtered LSRs (spread out over a larger area), and last I checked, OUN has only confirmed 3 tornadoes from the event so far, but they haven't put out an update in more than 24 hours which leads me to believe the surveys are done. Tulsa has only confirmed the Barnsdall tornado and expects to find more. Springfield has confirmed 3 (outside the high risk area).
Practically perfect obs:
Scaling is a bitch:
There are 421 filtered LSRs as of 10:45 AM MDT the day after this event.
Meanwhile, Monday's event still only has 355 filtered LSRs (spread out over a larger area), and last I checked, OUN has only confirmed 3 tornadoes from the event so far, but they haven't put out an update in more than 24 hours which leads me to believe the surveys are done. Tulsa has only confirmed the Barnsdall tornado and expects to find more. Springfield has confirmed 3 (outside the high risk area).
Last edited:
