Forecasting Accuracy 2024 - Good or Bad?

[Warren Faidley]

Lots of (off topic) discussions in other threads about forecasting issues in 2024.

Not intended to bash any entity, but hopefully a constructive discussion.

 

[Mark Gressman Jr]

Lots of watches and warnings the past few days with not much in the way of reports. I guess the main view is it's better to be safe than sorry when it comes to alerting the public. Any thoughts on why storms didn't produce? The components needed for severe storms were there, but I thought the risk levels seemed a bit ambitious if nothing else.

 

[Dan Robinson]

If we still have any readers from the SPC, I'm curious if there is a policy against downgrading outlooks after 1300z. There have been several examples of events where the environment very evidently seemed trashed by mid-morning (due to clouds/precip, an MCS scouring the moisture and shunting it south, or something else), yet the outlook remained at its 1300 maximum. The Elmer day in 2015 was like that, the overnight MCS scoured moisture down to the Red River, yet the higher tornado risk level in Kansas and Nebraska from earlier outlooks was maintained. I figured there was some social science component to this - maybe the impact to public trust in the forecasts is worse when a downgrade is made versus when the forecast busts? Again, I'm not bashing the SPC as they do an excellent job, and busts are just part of dealing with the weather using our current science and tools.

 

[Warren Faidley]

I was told once, years ago, that they don't like downgrading outlooks because NWS and many other Federal and State agencies adjust the number of people they staff based on outlooks. I do not know if this is true.

 

[Mark Gressman Jr]

I think you're on the right track Warren. I remember reading an older thread about SPC risk classification where Rich Thompson mentioned something to that effect. He stated that it had an impact to the prior planning of outside organizations such as EMs. He also touched on the gamble of lowering the risk classification and then having storms over perform, basically saying it would have to be a sure thing on the part of the forecaster.

Again, I'd imagine it's generally better to bust and have nothing happen than the the other way around. Since I've been tracking the weather, the SPC has done an amazing job... and I was surprised to see an otherwise adequate environment to my novice eye, fail to produce. Dew points seemed sufficient in the last few cases, but maybe the moisture wasn't deep enough. Shear was plentiful as well, but instability was marginal, as is to be expected with early season setups. If forecasting was easy, it wouldn't be as rewarding to nail a prediction.

 

[Noah Anderson]

He also touched on the gamble of lowering the risk classification and then having storms over perform, basically saying it would have to be a sure thing on the part of the forecaster.

To me, this makes the most sense--let's say the forecast downtrends from warranting a moderate-risk to an enhanced risk. How are you supposed to adequately convey to the public that they should still be alert for strong tornadoes, but that the expectation of an outbreak has passed? It's a catch-22 that could only promote lethargy on the part of the public when there may still be a relatively volatile environment in place. Also, imo, there is enough inherent uncertainty of any given forecast even the day / afternoon-of (e.g the All-High probs Tornado watch from May 20, 2019), that the marginal benefit of precision gained by performing such a downgrade is too small.

On the other hand, I too am perplexed by those cases that Dan mentioned, where e.g. the warmfront simply didn't reach as far north as expected, and yet they maintain a sig-risk area for tornadoes which is clearly north of the warm-front. This has always confused me.

 

[Mike Smith]

I see the mission of SPC as producing the most accurate forecasts possible, not directly (by, say, not downgrading) affecting personnel decisions.

 

[Jesse Risley]

I'm not sure what entities this is referencing, but from the media side, I'd argue this is still a consequence of this not being an exact science. I have to wonder if high-resolution model simulation, including CAMs, plays a role here. Nonetheless, despite all of the tools that have at our disposal, and some of the best minds to which those tools are applied, forecasting the behavior of a fluid dynamic is going to sometimes be erroneous. I tend to focus on the bigger picture here. Our lead times for watches and warnings are strides above what it was decades ago, so it's not like we're dealing with pyrrhic victories when these setups evolve as forecast. On the microscale, I haven't noticed anything particularly grossly inaccurate. In terms of downgrading higher-end outlooks, particularly with SPC forecasts, there are social science nuances involved that have already been discussed. That's difficult because we oftentimes don't know which mesoscale factors will enhance or reduce severe risk in a given area when 12+ hour lead times need to be given. I think most agencies do a decent job of conveying wording that expresses potential caveats or uncertainties with high-end events. We don't always have a way to accurately forecast when and where, for example, kinematics may be there for a substantial severe weather risk but thermodynamics don't align at the eleventh hour. I still believe it's better to be proactive and help the general public understand that, while there are caveats, the atmospheric ingredients exist to put their location in a potentially substantial risk of severe weather. We just don't have the science to affirmatively tell Whoville that a QLCS tornado will arrive at 2:30 PM on April 1st.

 

[Mark Gressman Jr]

I'd like to retract my previous statement about not having a lot of reports. Just went back and checked, and at least for the 9th, there are quite a few reports from what I'd imagine, are based on surveys. I think the easiest answer to my question of why storms didn't appear to be as severe as forecasted, was as Dan alluded to, overturning from prior convection and messy storm modes. I'd like to think that no one is complaining or unhappy with the outlooks. At least for me, I was more interested in getting opinions from those more knowledgeable than myself as to why the the last few outlooks didn't appear to pan out initially. I should've known better, and demonstrated a little patience in waiting for the final tally to come in. I'd also imagine storms down in the jungles would have less eyes on them to drive real time reports. Live and learn, that's why I'm here, lol.

 

[Matthew Crowther]

Here is a take based on some less-than-stellar forecasts for this severe season so far. Many forecasters, including ones who are far better than I are sometimes not paying enough attention to one very basic thing: Will there be any storms at all in the area were parameters look otherwise to be great for supercells/tornadoes? Conversely, in the same vein will there be *too many* storms? For an example of the former scenario, we need look no further than yesterday. In the area from SW Kansas to West TX the parameters like the shear/CAPE combo on forecast soundings looked to be pretty optimal for supercells/tornadoes. The rub, however, was there were only a handful of storms, and there were few severe reports and no tornadoes.

My concern all along for yesterday was that for several days, including leading right up to the event, the models, including the now pretty numerous CAMs, were very consistent in showing a rather large zone where no or little convection except for a zone in NW Texas. The attached frames are the 00Z Mon HREF ensemble forecasts for updraft helicity at 00Z Tue. Note the lack of any sort of signal in the KS/OK area. This is one snapshot of one set of models, but this trend was very consistent not just from various models, but it was also consistent over quite a few days before the event,

My general point is that forecasters need to look not just at how good the environment looks from a parameter standpoint, but they also need to examine several runs of various models to see if storms will actually occur, or in the latter scenario above, see if the storms might be so numerous that chasing will be not productive for seeing a photographable tornado or good storm structure.

 

[JamesCaruso]

Maintaining forecast continuity is definitely a consistent practice. Whipsawing the forecast back and forth just causes confusion and undermines credibility probably more so than a busted forecast. It’s an example of how precision can actually sometimes be the enemy of a forecast’s usefulness, relevance and accuracy - or, as Noah put it:

the marginal benefit of precision gained by performing such a downgrade is too small.

I think this comment by Noah was also insightful:

let's say the forecast downtrends from warranting a moderate-risk to an enhanced risk. How are you supposed to adequately convey to the public that they should still be alert for strong tornadoes, but that the expectation of an outbreak has passed?

I have seen explicit reference in NWS AFDs and in NHC discussions to maintaining continuity. For example, with winter storms here in Philadelphia it’s not uncommon to see reference to recent model runs backing off on snowfall totals, but the forecaster documents his/her explicit decision to defer changes to the forecast unless/until the model trend continues.

However, I have never seen SPC explicitly refer to maintaining continuity. I never see from SPC anything to the effect of, for example, “initiation is now in doubt, but we are maintaining the Enhanced Risk for continuity.” It almost seems like the discussion seeks to continue justifying the previous outlook, even while hedging with the wording. Yesterday 4/15/24 seems to be an example of maintaining continuity. There was a 30% risk, from at least Day 6. After a couple days of enhanced wording, the language softened, yet the risk area and level remained unchanged. It’s important to remember that convective outlooks are not really designed for the general public, and I assume the local NWS offices that rely on them understand these nuances. But I’m not sure about emergency managers or the media. Even during the day yesterday, there was an MSD for Nebraska that gave a 95% probability of watch issuance. No watch was issued, at least not within the ensuing four hours, but it’s not as if there was a subsequent MSD to that effect. That’s an example that might cause potential confusion for the audience of the product - what are they to make of the threat level as they wait for a watch that never comes?

As an aside, I find the concept of forecast continuity, the trade-off between precision and relevance/helpfulness of a forecast, and communicating uncertainty really interesting and I have applied it to my own profession as a CFO in communicating financial forecasts. Here’s a blog post I wrote for a professional journal if interested:

Page Error

www.picpa.org

 

[Dan Robinson]

I think a lot of this has to do with the relative infancy we're still in regarding the science of convective severe storms meteorology. There is obviously a lot we still don't know, and that is repeatedly evident time and time again. Add to that the coarseness of our surface and upper-air observations, and it's going to be an uphill battle. I wonder if we all are expecting too much of the enterprise? Could we have been spoiled by stretches of nailed forecasts for more classic setups? It does make me feel better about many of my busts when the best-of-the-best struggle this frequently.

 

[Brett Roberts]

Here is a take based on some less-than-stellar forecasts for this severe season so far. Many forecasters, including ones who are far better than I are sometimes not paying enough attention to one very basic thing: Will there be any storms at all in the area were parameters look otherwise to be great for supercells/tornadoes? Conversely, in the same vein will there be *too many* storms? For an example of the former scenario, we need look no further than yesterday. In the area from SW Kansas to West TX the parameters like the shear/CAPE combo on forecast soundings looked to be pretty optimal for supercells/tornadoes. The rub, however, was there were only a handful of storms, and there were few severe reports and no tornadoes.

My concern all along for yesterday was that for several days, including leading right up to the event, the models, including the now pretty numerous CAMs, were very consistent in showing a rather large zone where no or little convection except for a zone in NW Texas. The attached frames are the 00Z Mon HREF ensemble forecasts for updraft helicity at 00Z Tue. Note the lack of any sort of signal in the KS/OK area. This is one snapshot of one set of models, but this trend was very consistent not just from various models, but it was also consistent over quite a few days before the event,

My general point is that forecasters need to look not just at how good the environment looks from a parameter standpoint, but they also need to examine several runs of various models to see if storms will actually occur, or in the latter scenario above, see if the storms might be so numerous that chasing will be not productive for seeing a photographable tornado or good storm structure.

I agree with much of what you said in principle. I'm just not sure how applicable some of it is to the 2024-04-15 event.

I'm seeing a remarkable amount of Monday morning quarterbacking since about 6p yesterday, mainly on social media. To be clear, I respect everyone's point of view here on ST (including yours, Matt) because we're having reasoned discussion, unlike most of social media.

But my perspective here is that yesterday's lack of robust supercells across KS, the Panhandles, and W OK was hardly obvious beforehand. Many global models showed CI along the dryline. The ECMWF was fairly consistent in this, with others like the GFS and ICON also showing it on many runs. Sure, the overall tendency was for the event to begin around 00z, which is not great for chasing. But, to put it plainly, the world renowned ECMWF did very poorly with this event -- despite having a consistent depiction that would've instilled confidence, without the benefit of hindsight.

Combine the non-CAM model guidance with the fact that there were decent height falls projected by 00z, and concerning morning cloud cleared out rather nicely by early afternoon for KS and the N Panhandles... and I don't think the SPC outlook, or people's choice to chase, was unreasonable. In some ways, you could look at this event like an alternate permutation of 2019-05-17 where the cap verified just a bit stronger than forecasts, rather than the opposite.

Dryline setups like yesterday are inherently volatile in outcome, with everything riding on one tipping point. Close doesn't count in the outcome, but it should count in the evaluation of forecasts, IMO.

I may be biased and self-justifying since I was sitting in a gravel lot with my head hung at 8p yesterday, but that's my 2c.

 

[JamesCaruso]

I think a lot of this has to do with the relative infancy we're still in regarding the science of convective severe storms meteorology. There is obviously a lot we still don't know, and that is repeatedly evident time and time again. Add to that the coarseness of our surface and upper-air observations, and it's going to be an uphill battle. I wonder if we all are expecting too much of the enterprise? Could we have been spoiled by stretches of nailed forecasts for more classic setups? It does make me feel better about many of my busts when the best-of-the-best struggle this frequently.

Specifically with regard to chasing and only chasing, I actually hope forecasts, the related tools, and the observation network don’t get much better! The uncertainty is part of the appeal. I’m not a poker player, but I imagine it’s a similar inclination and personality type that enjoys weighing the blend of knowns, unknowns, probabilities, etc. and decision making in an environment of uncertainties. I don’t think chasing would be as satisfying if it was 100% successful all the time, instead of a more baseball-like .300 average. It’s the challenge that makes chasing fulfilling. Thinking of the recent eclipse, suppose the exact day, minute and path of an EF5 tornado were known in advance, and you could just show up to watch it (along with hundreds of others who had to do little more than look up the path and schedule online) - would you even want to chase anymore???

 

[gdlewen]

Thinking of the recent eclipse, suppose the exact day, minute and path of an EF5 tornado were known in advance, and you could just show up to watch it (along with hundreds of others who had to do little more than look up the path and schedule online) - would you even want to chase anymore???

For sure I would still chase if the batting average went up to 1.000. I would chase more. Busts are a serious drain on resources, which I did not appreciate until the economy began to go south in 2021-2022. In 2021 the idea of a two-day panhandle chase with an overnight stay was, while non-trivial, definitely doable. Not so much in 2022...and afterwards. Lots more same-day "there and back again" excursions. That's why I spend so much time trying to figure out on which days to chase, and when things go wrong, why they went wrong. To optimize the consumption of resources.