Forecasting Accuracy 2024 - Good or Bad?

[Jeff Duda]

I'm gonna continue to pile on here.

Yesterday technically would have verified a high risk for wind (SIG is required in addition to 60% coverage, and there are a small number of sig wind reports within the 60% PP contour). The 60% PP level was also exceeded for hail, but it seems that there is no hail coverage threshold that is adequate to satisfy a high risk according to SPC's standards, so for the time being I'll let them have that.



Without loss of generality (i.e., all of the forecasts maintained the same max category), here is the day 1 convective outlook from yesterday:


The highest category was enhanced - that means this forecast is a two-category underforecast. That also makes arguably (see edit to my previous post) two forecasts this week that missed low. Yet for some reason these two underforecasts aren't making the headlines like the overforecast was.

 

[Jason N]

that's an interesting point you bring up. Based on what you're pointing out, it seems that they "reserve" the High for tornado days only? I wonder how many times they used high for Derecho events. Maybe that's how they view it then. Tornadoes, Derecho get the top spots on High Risk days?, while MCC's or long-lived wind events get enhanced, maybe MDT if conditions warrant a higher tornado prob? .. I don't know but that's interesting.

 

[Jeff Duda]

that's an interesting point you bring up. Based on what you're pointing out, it seems that they "reserve" the High for tornado days only? I wonder how many times they used high for Derecho events. Maybe that's how they view it then. Tornadoes, Derecho get the top spots on High Risk days?, while MCC's or long-lived wind events get enhanced, maybe MDT if conditions warrant a higher tornado prob? .. I don't know but that's interesting.

No, higher-end derechos are deserving of high risk designation, too, in addition to tornado outbreaks.

 

[Andy Wehrle]

I think part of the reason we don't see high risks for 60% hatched wind more often is that the truly exceptional, top-end derechos (the ones with widespread >100 MPH gusts over a long swath) are hard to forecast with accuracy. The most recent such event (the August 10, 2020 Iowa-Illinois derecho, which IIRC produced at least one 120 MPH measured gust and one estimated at 140 MPH [category 4 hurricane!] based on damage) had a marginal risk covering most of the eventual affected area on the initial Day-1 outlook. When it became apparent what was about to occur, they upgraded to a 45% hatched (MDT) for wind and issued several PDS severe thunderstorm watches.

The few times we have seen those risks issued (June 3, 2014 for example), the resulting event has been relatively underwhelming (and in that case, the core of the event tracked further south with the majority of the high risk zone void of severe reports).

 

[John Farley]

Does it seem odd to anyone else that there were no watches at all yesterday (June 5)? I was not paying much attention to the weather yesterday, so I don't know much about parameters that were in place or what the models were showing, but based on storm reports I would think that there should have been watches of some kind in the MI/OH area and also for MD and nearby areas. There were numerous tornadoes in MD and the 4th largest number of TOR warnings ever in one day for that state yesterday (22), according to TWC. And 2 tornadoes and other severe reports in Michigan, including a tornado that caused a fatality in the Detroit suburbs. Am I missing something?

 

[Mike Smith]

Does it seem odd to anyone else that there were no watches at all yesterday (June 5)? I was not paying much attention to the weather yesterday, so I don't know much about parameters that were in place or what the models were showing, but based on storm reports I would think that there should have been watches of some kind in the MI/OH area and also for MD and nearby areas. There were numerous tornadoes in MD and the 4th largest number of TOR warnings ever in one day for that state yesterday (22), according to TWC. And 2 tornadoes and other severe reports in Michigan, including a tornado that caused a fatality in the Detroit suburbs. Am I missing something?

You aren't missing a thing. It is especially odd since meteorological Joe Bastardi, the day before (June 4), predicted that SPC would forecast an "enhanced" risk of tornadoes for yesterday. NADOCAST pretty well nailed it.

If you look at the VAD winds and ACARs soundings, it was a pretty classic setup. While I do not know any specifics of what was going on at SPC yesterday, I do fear that mets these days pay too much attention to models and not enough to monitoring the real-time data.

 

[Jason N]

I have a good friend of mine, Gerard Jebaily (WBFF Fox45 weather) and Justin Berk who I spoke with on the issue. There was a short MCD on the outying tornado threat for the area, with a 20% prob of a watch, I think they'll need to revisit it to figure out what was happening synoptically. Possibly some kind of MCV that enhanced the parameters, but oddly enough on the tail end of a significant amount of precip. I hadn't noticed it until it was already happening, but I know there was not watch ever issued, only the MCD with a Marginal in place all day.

 

[Mike Smith]

As is often the case with low-end events, NADOCAST nailed it.

 

[John Farley]

Jason, if you are talking about MSD 1173, I believe that it barely included the area where the fatal tornado occurred, did not include areas farther north than that where severe weather occurred, and was for a period earlier than when the aforementioned tornado happened.

 

[Jason N]

Jason, if you are talking about MSD 1173, I believe that it barely included the area where the fatal tornado occurred, did not include areas farther north than that where severe weather occurred, and was for a period earlier than when the aforementioned tornado happened.

John ,

I was referring to #1177, the earlier 1175 did speak to an outlying risk of tornadoes, but as you highlighted, not in the same geographic area, which the later MCD 1177 did speak more to and adjusted that area north as the risk area moved as well. I was probably off saying MCV being partly the cause over a Short Wave Trough, but again, I wasn't really looking at the area intently.

 

[Jeff Duda]

That could have possibly been a policy issue. I'm not sure. But the predictability and potential also was poor and low, respectively, IMO.

I checked the mesoanalysis archives and the SCP only reached 4 in localized areas of MI and across the mid-Atlantic until it bumped up to 8 only over DC for one hour at 23Z. Keep in mind that such an analysis would not have been available to SPC forecasters until probably 2330 or so, and the tornado event was half over by then.

Although the MD area tornado swaths appeared to be a bit of an "overperformance" by the atmosphere, I question the predictability of this event even an hour or two beforehand. HRRR forecasts initialized just a few hours before the event gave little indication of serial tornadoes across the region. The environment, despite technically exceeding the low-bar threshold of "1", probably didn't appear particularly conducive for any kind of concentrated severe event, which is probably a requirement for a weather watch. In the absence of confidence on a location, that would explain the lack of a watch.

As an example, here is the 22Z HRRR forecast valid at 00Z:

Sure, in hindsight it's easy to look at the DC area and go all Leo DeCaprio "OOH! See! The HRRR is highlighting the DC area! Should be a watch!"
Okay, but there are similar levels of SCP over SE MI/NW OH with no tornado reports (the Livonia tornado occurred several hours prior to this). And there's an even higher SCP spot of about the same size in C WI associated with no tornadoes and only scattered severe wind reports.

It's not easy to look at an existing or forecast quasi-marginal environment scattered and spread about a large region and be able to narrow down the corridors where severe will occur. And I think the spirit of weather watches is to highlight a smaller region where more focused severe is likely. Right or wrong in hindsight, I can see why a forecaster may have decided not to flip the switch on any watches given this is what they were able to see at the time.

Finally: if your primary goal is POD = 1, then you should be issuing watches all over the place all the time from 1 June - 31 Aug. But that is not realistic, and research has shown that tornadoes, in particular, do not behave the same way in summer environments in which the composite indices are high compared to how they behave in springtime environments with the same composite index values. And POD = 1, FAR =1 is not a desirable place to be - that is "no skill" territory.

 

[Jeff Duda]

As is often the case with low-end events, NADOCAST nailed it.

It got the DC area for sure, but it had false alarms for S MS, the Louisville area, and WC WI. If evaluated against PP obs, the brier skill score for that forecast would probably be negative.

 

[Jason N]

I think that's pretty reasonable Jeff from that standpoint. There will be days, that like you said, the atmosphere will just exceed expectations based on some unexpected/forecasted local series of phenomena that came into phase, but it will be interesting none the less to go back and dig into some of the analysis leading up to it. I do wonder what their policy is for issuing a watch based on real-time conditions on those days where the atmosphere has exceeded expectations is. Like you said, it was already too late at some point, so are the MCD's enough to compensate and let the NWS/WFO's handle it with SWS's and local warnings? it seems as though this may have been built into it perhaps, something akin to a conditional PACE plan.

 

[Jason N]

Sure, in hindsight it's easy to look at the DC area and go all Leo DeCaprio "OOH! See! The HRRR is highlighting the DC area! Should be a watch!"

I hope you spit a lot when you typed that. lol

 

[Mike Smith]

It got the DC area for sure, but it had false alarms for S MS, the Louisville area, and WC WI. If evaluated against PP obs, the brier skill score for that forecast would probably be negative.

Even when you consider that it reached a 5% threshold for DC and none of the others?

Regardless, Joe Bastardi nailed it the day before.

My point is that yesterday was hardly impossible to forecast and that it certainly could have been nowcast better than it was.