Forecasting Accuracy 2024 - Good or Bad?

[Jeff Duda]

My point is that yesterday was hardly impossible to forecast and that it certainly could have been nowcast better than it was.

Perhaps we need to just meet in the middle a bit or agree to disagree. My point is that, based on the best available data we had at the time, yesterday would have been a very difficult forecast, butting up right against the limit of practical predictability.

That one person happened to "nail" it (please, show me the forecast so I can have a look at it) should be regarded more as serendipitous than some kind of underlying skill (unless Joe Bastardi has a rigorous track record of properly forecasting every event; provide evidence if so).

This gets at ensemble/probabilistic forecasting. Generate a 1000-member ensemble centered on the HRRR forecasts of that event and it's likely that one of them will happen to hit it, but probably not more than a handful of members would. But those members aren't based on some kind of higher skilled underlying NWP model (if you assume that IC perturbations are used to form the ensemble, but without loss of generality it wouldn't matter even if you did use a multi-physics ensemble, too). It was more chance than skill in that case.

To cite a recent practical example on this site, I recently posted about Paul Markowski's talk on tornadogenesis mechanisms here: Updates on Thoughts Regarding the Mechanisms of Tornadogenesis (NCAR talk). The talk was based on an ensemble of idealized tornado simulations in which the ensemble diversity was provided solely via the random number seed used to generate initial perturbations that were then allowed to develop into turbulence structures. Yet some simulations developed long-track, strong tornadoes, whereas others produced only short-lived and weak tornadoes. All from a difference in a random number upstream of the simulation workflow! If a real-data case actually occurred such that there was a long-tracked EF3 tornado, no one would have been able to guess which of the forecast member(s) would be right in advance of the event. That means the member(s) that forecast the EF3 tornado mostly got it right by chance, which I'll admit is a kind of disguised way of saying "factors so small in scale and so transient, and also of such high order, that no human has the capacity to track such factors forward to making a forecast that is guaranteed to capture the observed event."

 

[JamesCaruso]

Does it seem odd to anyone else that there were no watches at all yesterday (June 5)? I was not paying much attention to the weather yesterday, so I don't know much about parameters that were in place or what the models were showing, but based on storm reports I would think that there should have been watches of some kind in the MI/OH area and also for MD and nearby areas. There were numerous tornadoes in MD and the 4th largest number of TOR warnings ever in one day for that state yesterday (22), according to TWC. And 2 tornadoes and other severe reports in Michigan, including a tornado that caused a fatality in the Detroit suburbs. Am I missing something?

Thanks for flagging this John. I wasn't really looking at things yesterday (June 5) because I was traveling back from Dallas to Philadelphia (flight delayed three hours and a baggage claim nightmare due to late equipment change, the Philadelphia airport was a nightmare just because of some non-severe thunderstorms). I did notice four tornado warnings in Maryland when I peeked at radar but the situation didn't really register with me during all the travel shenanigans. My "home" NWSFO of Philadelphia/Mt. Holly handles part of Maryland, so I'll be interested to learn what I can about this!

EDIT: I hadn’t looked closely enough when I wrote the above. The area in MD where the tornados occurred is covered by the Baltimore/Washington NWSFO, not Philadelphia/Mt. Holly, which I think covers only the Delmarva portion of MD.

 

[Mike Smith]

Jeff,

I hope I am not giving the wrong impression, I have a great deal of respect for your work and for you, personally. If you had asked me 15 years ago, I never would have guessed we would come so far, so fast with CAMs. I apologize for anything I may have written that gave another impression.

That stated, please read this: June 8, 1974: "The Day Television Weather Grew Up"

I have attached the (then) state-of-the-art "baroclinic model" from June 7. There was no surface feature forecast, no forecast of stability, nothing beyond what you see... just 500mb vorticity and heights. We had no satellite imagery of any kind. There was no upper air data between 12 and 00Z.

What did we have? There were techniques, taught in college synoptic labs, to handle those elements of severe thunderstorm forecasting. Colonel Robert Miller's USAF tornado and severe thunderstorm manual was excellent.

We made a highly skillful forecast for on June 7, and the morning of June 8, for the severe storms the afternoon of the 8th, at WKY. We had a policy of not forecasting tornadoes and severe thunderstorms the day before. NSSFC's outlook was only 24 hours from 12Z to 12Z. I discussed what to do with one of the long-time WKY people and he said, after I reviewed the 7p (00Z) data Friday evening, "If you are that confident, you should tell the viewers." I did, on the 10p weathercast.

Next morning, NSSFC went "Numerous*" over Kansas and Oklahoma for the first time (in their history, I believe). During my coverage of the pre-dawn severe thunderstorms, I talked up the threat for that afternoon. How did this occur?

I can't speak for the late Hilmer Crumrine who made the 8am June 8 NSSFC severe weather outlook, but I can tell you that I plotted two surface charts that morning and did my own 500mb and 850mb analysis. I did a 1pm surface chart and saw a strong mesolow on the dry line just northeast of Lawton that was advancing rapidly east. Steve Tegtmeier had done his thesis on severe weather pattern recognition and had discovered that northeast of a mesolow was/is a favored area for tornadoes. I had that in mind, and it influenced my messaging, during my on-air bulletins between plotting that chart and the formation of the hook southwest of KOKC.

While I certainly concede this wouldn't have been possible more than about 36 hours in advance, it demonstrates humans can produce skillful severe thunderstorm forecasts by adding analysis, pattern recognition, etc. to the information provided by models.

Perhaps it is a generational issue, but I believe forecast and applied weather science would be enhanced if we reincorporated some of these ancient -- but still quite reliable -- techniques. It wouldn't be a bad idea to teach a distilled version of Miller's techniques in today's forecasting classes.

Best wishes,

Mike

* In that era, NSSFC's 24 hr severe thunderstorm forecast was stated as, isolated, few, scattered, numerous.

 

[JamesCaruso]

Adding another forecasting anecdote - I thought both June 3 and June 4 were interesting.

See my post about June 3 here: 2024-06-03 EVENT: TX/OK

"We stopped around 7:20 to consider our next move, when an SPC MSD discussion for the area popped up on RadarScope - #1156, issued at 7:22pm CDT - "...latest hi-resolution radar imagery from Amarillo shows multiple isolated severe storms ongoing across the central and eastern Texas Panhandle." WTF?!? There were NO ongoing storms or warnings at that time! Clearly the MSD was drafted a little earlier, but there were never more than two or three storms, which I suppose technically is "multiple" but in my view the MSD way overrated the situation as of the time it was issued.

It was clear we were capped out (pun initially unintended, but now intentionally left upon editing ), and even NWS Norman in their evening AFD said their focus on severe potential had now shifted to overnight development in central OK. Surprised that overcoming the cap, especially while conceding that large-scale forcing was absent, was never mentioned as a potential failure mode by SPC or NWS Norman."

On June 4, SPC seemed bullish on storms going up by 6:15 (per an MCD), never mentioning potential cap issues. In fact, they explicitly mentioned "small" inhibition in an early outlook (NWS Norman, however, did mention potential capping issues). We're still sitting out there with struggling/dissipating storms at around 7pm. At least the skies weren't completely blue. A later MCD after 7pm did mention storms struggling with the cap observed on the 7pm sounding, but that was the first such mention if I'm not mistaken.

 

[Jeff Duda]

Jeff,

I hope I am not giving the wrong impression, I have a great deal of respect for your work and for you, personally. If you had asked me 15 years ago, I never would have guessed we would come so far, so fast with CAMs. I apologize for anything I may have written that gave another impression.

That stated, please read this: June 8, 1974: "The Day Television Weather Grew Up"

Next morning, NSSFC went "Numerous" over Kansas and Oklahoma for the first time (in their history, I believe). During my coverage of the pre-dawn severe thunderstorms, I talked up the threat for that afternoon. How did this occur?

Thank you for the kind words, Mike. I read your article. You certainly have an impressive repertoire of service in meteorology and I'm glad this forum has a primary source of pioneering severe weather warning media. I can't imagine too many other websites contain such a high-status individual.

Regarding your question of how you were able to make such a good forecast based on so little information -- I would guess the synoptic scale pattern recognition was the biggest factor contributing to your forecast. However, a simple forecast of "numerous" (I'm guessing that refers to severe storms) over two states is appropriately vague given the modeling ability and forecasting techniques of the time. I'm also guessing that you would not have had much confidence if asked to pick sub-state regions within the broader KS/OK region that would see a higher threat than surrounding areas. But that's what today's CAMs now generally offer - the ability to focus on smaller regions so that humans don't have to alert entire states to the potential for severe weather.

It is generally well known that the temporal forecast horizon increases with scale -- synoptically evident events are inherently more predictable further in advance compared to synoptically-weakly-forced events (i.e., mesoscale driven events). The event you speak of appeared to be strongly synoptically forced with clear-cut troughs and surface boundaries. There was a trough upstream of yesterday's event, although with a bit of spacing to the mid-Atlantic. Also, WPC's surface analysis only showed a stationary front nearby, and north of where the torndoes occurred. Finally, the forecasts of the mesoscale environment (the composite indices) indicated that severe storms were possible in a very broad region. Again, while some HRRR forecasts did actually pinpoint the MD/DC area (props), at the same time it also pinpointed other areas where focused severe weather did not appear to occur. Anyone basing a forecast off that would have had some false alarms to go with their hits.

 

[Mark Gressman Jr]

Being that I live in the general area of the Maryland storms, I had a very close eye on what happened as it developed. While I'm far from an expert, I certainly didn't anticipate those storms producing. Storms started out as low topped multicellular clusters with no rotation as bulk shear was mostly non existent. There was a decent amount of CAPE for the area and hodographs had good curvature, but the small amount of helicity was confined 0-500M. Once the first tornado was in progress, several other cells almost immediately developed hooks with weak velocity couplets. At that point, I agree with John in that I was perplexed that no watch was being issued. A mesoscale discussion came out shortly after, making mention of the likelihood of future tornadoes being small, as instability was expected to diminish rapidly with the onset of evening. The SPC had that area marked as a 2% on the Day 1 outlook, so I don't think it was a complete surprise... but from the video I saw of one of the tornadoes, the impressive vertical motion was not what I would've expected from that particular setup.

 

[Dan Robinson]

The Maryland event is a good example of why I have my pathological home area bias that has cost me tornadoes in both the Plains and the Midwest. You can have 50 such setups that look identical synoptically and do nothing, and then one inexplicably goes nuts like this. As I mentioned earlier in this thread, I think it is beyond our current science and understanding to reliably discern the ones of this type that end up producing. No one can do it yet - not the SPC, not the NWS, not models, not Nadocast, not the most experienced chaser.

For me as a chaser, the only way I can avoid missing that one big day like that here is to chase every single one that has those basic ingredients (which at least here, is a warm/stationary front with any CAPE and 30kts+ midlevel flow). That has meant that I choose staying here (or coming back) every time those ingredients are present, the cost to me of which is missing big events like the Eldorado/Duke tornado. For the SPC, doing the equivalent of that (drawing a 5% or issuing a watch for every setup like this) would catch the one that does produce, but at the cost of an exorbitant FAR.

I don't think this problem is going away any time soon. I think the forecast busts (in both directions) are here to stay until some type of breakthrough is made in both 1.) sorting out those subtle mesoscale factors that make or break tornadic vs non-tornadic, and 2.) a way to measure or derive those factors from either the observations we already have or via new methods of sampling the atmosphere. In the meantime, as a chaser at least, your only option is to chase them all if you want to catch the one that goes big.

 

[Boris Konon]

Being that I live in the general area of the Maryland storms, I had a very close eye on what happened as it developed. While I'm far from an expert, I certainly didn't anticipate those storms producing. Storms started out as low topped multicellular clusters with no rotation as bulk shear was mostly non existent. There was a decent amount of CAPE for the area and hodographs had good curvature, but the small amount of helicity was confined 0-500M. Once the first tornado was in progress, several other cells almost immediately developed hooks with weak velocity couplets. At that point, I agree with John in that I was perplexed that no watch was being issued. A mesoscale discussion came out shortly after, making mention of the likelihood of future tornadoes being small, as instability was expected to diminish rapidly with the onset of evening. The SPC had that area marked as a 2% on the Day 1 outlook, so I don't think it was a complete surprise... but from the video I saw of one of the tornadoes, the impressive vertical motion was not what I would've expected from that particular setup.

The MD tornadoes sure surprised me. I noted on the 13z SPC update, a small area of MRGL was added to DMV, and 2% tornado risk, but I did not think twice about it. I admit I was not paying close attention (working the overnights currently) b/c it was clearly not a "synoptically-evident" day. So I went bed late morning, only get up mid evening and retroactively realized what happened (long-tracked, tornadic supercell passed not more than about 7 mi N of me)!

The storms rode a warm front, so that explains a lot. Also, it was classic TC tornado-type environment (w/o a TC)! Low CAPE, high low-level shear, almost no diurnal heating, PWATs high, and poor lapse rates through the depth of the column. Storms had little lightning, and no large hail reports and only 4 straight-line wind reports. The number of tornadoes exceeded the hail/wind! You don't see that very often.

The event was so localized and short, one could agrue it fell below the criteria for a watch? Also, coverage of all severe wx reports were limited, so the MRGL risk could be said it verified.

 

[Mike Smith]

There was plenty for a watch. The perfect prog indicated it should have been "High" risk.

 

[Jeff Duda]

There was plenty for a watch. The perfect prog indicated it should have been "High" risk.

That has nothing to do with the topic being discussed. That's an a posteriori argument. We're discussing the signs that were apparent before the event occurred, not what should have been forecast given the known results.

 

[Robert S. Thompson]

...

For me as a chaser, the only way I can avoid missing that one big day like that here is to chase every single one that has those basic ingredients...In the meantime, as a chaser at least, your only option is to chase them all if you want to catch the one that goes big.

Chasing is easier than it used to be, but in the end it's still hard. You are essentially operating at the small end of the mesoscale at a level beyond the current tools. Even if the watches and outlooks are perfect, it's doubtful that storms and/or tornadoes will be uniformly distributed within the outlook/watch area (you can't chase an entire watch), and even if they were, the odds that you are in the right place at the right time with the right lighting and road network and even lower. Every year on my chasing trips there are days that overperform and others than bust from a large perspective. There are also days that were personally great for me because I was in the one place that performed, and days that were great for most folks but for whatever reason I didn't see anything good. This is why I just go out for a week or two, chase everything, and hope for the best. A beautiful storm, lightning show, or tornado makes the bad days worth it.

 

[JamesCaruso]

We're discussing the signs that were apparent before the event occurred, not what should have been forecast given the known results.

Chasing is easier than it used to be, but in the end it's still hard. You are essentially operating at the small end of the mesoscale at a level beyond the current tools. Even if the watches and outlooks are perfect, it's doubtful that storms and/or tornadoes will be uniformly distributed within the outlook/watch area (you can't chase an entire watch), and even if they were, the odds that you are in the right place at the right time with the right lighting and road network and even lower. Every year on my chasing trips there are days that overperform and others than bust from a large perspective. There are also days that were personally great for me because I was in the one place that performed, and days that were great for most folks but for whatever reason I didn't see anything good. This is why I just go out for a week or two, chase everything, and hope for the best. A beautiful storm, lightning show, or tornado makes the bad days worth it.

Well-said Robert. Every time I see someone say chasing is “easy” because of radar, CAMs, SPC tornado risk areas, whatever, it really irritates me, because it either means (a) the writer is clueless, (b) the writer is arrogant about their own success, or (c) I just really suck. While (c) may still be true, I have come to believe success in storm chasing is like baseball, a .300 average is pretty good, and my .250 or .275 may not be quite as bad as I thought. I always feel self-doubt when I find out what I missed, but that gets back to Jeff’s point - it’s ridiculous to judge one’s performance with the benefit of hindsight.

I do not necessarily chase *everything* while I’m out, especially if it’s a >4 or 5 hour drive. Then I kick myself when I miss something great, but given low probabilities you have to judge yourself with the lens of, “would I have been willing to make that drive 10 or 20 times to see that?”

It’s also hard not to experience self-doubt when you find out what others saw, but there is always going to be someone that did better *that day.* Nobody does better *every day.*

 

[Dan Robinson]

James and Robert, exactly! It’s pretty easy to learn the basics and even advanced tornado forecasting, but still very hard to get it right the majority of the time. When the SPC can’t get it right all the time being arguably the most qualified out there, we can’t expect more of ourselves.

I’m not afraid to admit that once I go not very far beyond the basics, I have no idea what I’m doing! I just hammer out enough chase days that I can see some good stuff with the resources I have and the success rate I feel I’ve topped out at.

More impressive to me than someone who chases everything and sees nearly every tornado is someone who can turn their back on the ones that have the tornado synoptics yet *don’t* produce. As in, they stay home/take a down day for those and *never* miss a big event as a result. I don’t yet know anyone who can do that.

 

[Jason N]

I think people can beat themselves up about it too much. I mean, if you really want to do that, factor in the cost of each tornado or lack there of that you caught in a season, in hotels, gas, food, rentals, (for those that do). if you spent 3K with zero success, I mean that's a pretty easy way to be like, I'm done.. or 3K for 2 tornadoes, 1500 per tornado!?! , but I personally don't see it like that. it's like going to the casino with money you know you're probably going to lose, or, you might win!, but its disposable for your enjoyment no matter what and you knew that going into it, or at least one would hope you do.

In terms of accuracy, all of the tech out there in models and data give even the average person a pretty fair chance at seeing "something", those with more experience or technical knowhow probably up their anti for a successful chase, but even the best of us lose out. So I never saw chasing as winning/losing. I think the biggest thing is I wish I had more time to do it. Those who do are the real winners IMO lol. I'll be a winner someday, but this damn thing called work has me shackled for another dozen years or so. Until then, pick my annual window and hope for the best, enjoy the moments and restaurants, sunsets, and scenery.

OH!! , speaking of restaurants, I got a couple to put into that thread!

 

[JamesCaruso]

I think people can beat themselves up about it too much. I mean, if you really want to do that, factor in the cost of each tornado or lack there of that you caught in a season, in hotels, gas, food, rentals, (for those that do). if you spent 3K with zero success, I mean that's a pretty easy way to be like, I'm done.. or 3K for 2 tornadoes, 1500 per tornado!?! , but I personally don't see it like that. it's like going to the casino with money you know you're probably going to lose, or, you might win!, but its disposable for your enjoyment no matter what and you knew that going into it, or at least one would hope you do.

I was ready to quit twice during my recent trip. I felt that way back in 2019 also. But after 28 years, I highly doubt I will ever follow through on it. While I still struggle not to get really frustrated at my own mistakes, the rest of the failure modes (the weather itself, road network, etc.) don’t bother me as much anymore. I have come to realize that I love the *process*, which doesn’t mean it’s always fun, but you have to “embrace the suck” like in any challenging endeavor.

I often think about what other hobbies might scratch whatever the itch is that storm chasing scratches… I actually think you hit on one with the gambling analogy. No, I’m not about to become one of those expressionless people that put coins into slot machines like zombies at all hours… But poker with friends might be interesting, with its mix of strategy/skill and randomness/chance that seems similar to storm chasing. I have never played before, but thinking of learning!