Objective scoring of previous chase seasons

[Brett Roberts]

With no storms and little to cheer us up on the models, I've been channelling some of my SDS the past few weeks into something semi-productive. There's always a ton of speculation about the upcoming season around this time of year, and some of that naturally involves drawing comparisons to previous years (analog forecasting, if you want to get fancy).

I recently realized I too often rely on anecdotes to judge the quality of previous chase seasons when looking at analogs; and even then, I can only remember so far back. So, I've attempted to create an objective scoring system for previous chase seasons that relies on official storm reports (retrieved as CSV files from SPC's excellent SeverePlot page). This dataset goes all the way back to 1951, as does my scoring system.

Here's the link: http://skyinmotion.com/weather/chase_season_rankings/

I won't bother trying to describe the methodology in detail here, other than to say the scoring is based primarily on tornado reports (with small weighting also given to giant hail reports) and that it strongly rewards activity spread out over many days in a season/month, as opposed to just one big outbreak day with tons of reports. All the gory details can be found here, if you're so inclined.

Scores/rankings are available not only for the Plains as a whole, but also for numerous subregions and individual states. Note that this is focused only on "Chase Alley" and does not include secondary chasing areas like Dixie Alley, Illinois, etc.

If you have any questions, or better yet suggestions for improvements, fire away! I hope at some point to look at score correlations with various combinations of indices, but that's a bigger project for sometime in the future. For now, it's just a nice quick-glance reference for the level of chaseable activity in previous years when looking at analogs.

 

[Adam Lucio]

Thats pretty creative. I like it. Though Im scared to see that, based on this, its possible to have a year worse than 2012. (You should omit 4-14 and see where it ranks then lol)

 

[Jon Strebler]

I like the weighting system that you used. Having a big outbreak doesn't make a chase season(ex. 2012) and that is reflected in how you scored the years. I'm not sure how it would skew the results, but have you thought about putting any weight based on an individual storm? If one storm produced multiple photogenic tornadoes then that can have an impact on how a season is judged to some degree, at least in my opinion. Rozel last year comes to mind. Mainly, I wonder how you can judge a storm from a chase perspective. A tornado with a path length >15mi sounds great and should be weighted highly in most cases, but what if that tornado occurred after dark? Should that tornado be given the same weight as one of equal length that occurred in the evening and was highly visible?

 

[Mike Johnston]

If I'm not mistaken from reading your methodology, you by-and-large excluded night tornadoes using time of day, so that should take care of part of Jon's concern. Beyond that, I'm not sure how you could make further adjustments for visibility given the database.

Also, interesting how you adjusted for report inflation over the years. By using 50% of roughly estimated inflation, it's almost like you used a Bayesian statistical adjustment which is probably as good could be done. That's a tough one.

 

[chrisbray]

I'd like to see this for Illinois as well!

 

[Jon Strebler]

I completely overlooked the fact that you already took away nighttime tornadoes.

 

[Dan Robinson]

Very cool, thanks for posting. I'm with Chris, Illinois would probably show some surprises!

 

[Andy Wehrle]

Interesting that 2005 ranked significantly higher than 2004 for overall season. If you'd ask the average chaser which season was better for them, probably 8 out of 10 would say "2004." I remember 2005 having a quiet May followed by a lot of June events, but for those you had to be (a) still able to get to the Plains and chase and (b) on the right storm or two. 2006 is no surprise to me based on my recollections, but I'm surprised 2009 is as high as it is. I'm also surprised 2011 isn't higher, but in retrospect that's not so much of a surprise since that year's explosive April was largely concentrated east of the Plains.

 

[Jeff Duda]

Very cool, Brett. Interesting to take a look at things. I have two comments/questions:

1) Why did you decide to give less weight to the number of days with a tornado of path length .ge. 15 mi. than to the number of days with a tornado path length .ge. 4 mi.? Seems sensible to give the former more weight since chasers would probably consider a season to be better if they got to chase a bunch of long-tracked tornadoes rather than medium/short-tracked tornadoes.

2) Have you considered performing the linear regression based on tornado counts with (estimated) ratings of (E)F0 and (E)F1 rather than final relative score? I seem to recall that the majority of the increased counts of yearly tornadoes over the years has been due to more low-end tornadoes spotted/reported. I suppose, however, your current method also kind of takes into account the increased availability of chasing since the 1950s. I'm guessing it wasn't even possible for as many people to get out and traverse the central US searching for tornadoes back then with the lesser technology (or maybe the technology was fine and the reasons were more societal and I'm just wrong).

 

[Brett Roberts]

Illinois has been added by popular demand! Just keep in mind that its stats do not contribute to any of the larger regions. Also, I didn't bother trying to define a chaseable portion of IL, so reports from the entire state are used in the calculations.

I like the weighting system that you used. Having a big outbreak doesn't make a chase season(ex. 2012) and that is reflected in how you scored the years. I'm not sure how it would skew the results, but have you thought about putting any weight based on an individual storm? If one storm produced multiple photogenic tornadoes then that can have an impact on how a season is judged to some degree, at least in my opinion. Rozel last year comes to mind. Mainly, I wonder how you can judge a storm from a chase perspective.

I would love to find some way of doing this with the dataset we have. So far, I haven't thought of one, but perhaps I'm not thinking creatively enough. One halfway-feasible way of judging storm "quality" would be to take into account storm motion (penalize fast-moving, reward slow-moving). But even this does not seem to be possible if you want to use data all the way back to the 1950s; tornado duration isn't recorded, and trying to use hail reports as a proxy would likely fail in early decades, when hail reports were far less common.

Interesting that 2005 ranked significantly higher than 2004 for overall season. If you'd ask the average chaser which season was better for them, probably 8 out of 10 would say "2004." I remember 2005 having a quiet May followed by a lot of June events, but for those you had to be (a) still able to get to the Plains and chase and (b) on the right storm or two. 2006 is no surprise to me based on my recollections, but I'm surprised 2009 is as high as it is. I'm also surprised 2011 isn't higher, but in retrospect that's not so much of a surprise since that year's explosive April was largely concentrated east of the Plains.

As I was looking at the scores initially, I had some of the same thoughts. I came to the conclusion that subjective impressions by real storm chasers will probably always trump any objective measures, at least until we have datasets that can somehow be used to derive something like "photogenic quality!"

It's probably best to think of these scores as measuring the activity level of a season, and not so much the quality, if that makes sense. Given my intention is to use the scores in conjunction with analog forecasting, I feel okay about that. In general, higher scores indicate more persistently-favorable patterns for chasing, which was my goal. I think there are years like 2004 when everything seems to go right for chasers in a mesoscale and/or storm-scale sense, and I'd agree that it was the best chase year of the 2000s. On the other hand, if I'm trying to forecast the 500 mb pattern for this upcoming spring by comparing indices with previous years, I'm hoping to see similarities with years that persistently produced storms/tornadoes, regardless of how chasers viewed them subjectively. In my mind, a consistently storm-favoring pattern gives us the best chance of having a great season; the smaller-scale details (i.e., things that determine chaseability) will work themselves out when the favorable setups arrive.

Also, just a general reminder that you can hone in on sub-regions using the drop-down menus. In the case of 2004 vs. 2005, it does seem very counter-intuitive. It turns out that the northern Plains lit up big-time in June 2005, which is having a big effect on the score. So if you look at the "Southern/Central Plains" category, things look a little more reasonable, from the perspective of someone who doesn't venture much into ND/MT/WY.

 

[Shawn Schuman]

2) Have you considered performing the linear regression based on tornado counts with (estimated) ratings of (E)F0 and (E)F1 rather than final relative score? I seem to recall that the majority of the increased counts of yearly tornadoes over the years has been due to more low-end tornadoes spotted/reported. I suppose, however, your current method also kind of takes into account the increased availability of chasing since the 1950s. I'm guessing it wasn't even possible for as many people to get out and traverse the central US searching for tornadoes back then with the lesser technology (or maybe the technology was fine and the reasons were more societal and I'm just wrong).

Yeah, that's where most of the inflation comes from. Actually you could make the case that significant tornado numbers may have been inflated somewhat in the past compared to today's standards.

Anyhow, that's a really interesting idea Brett. Correlating chase seasons with indices sounds especially intriguing, though I'm sure it'd take quite a while. I suppose what you're really measuring is the opportunity for chasing. That may not translate to quality chasing, but it's handy to have some measure of the opportunities available regardless of how they turned out for chasers in practice.

 

[Brett Roberts]

Very cool, Brett. Interesting to take a look at things. I have two comments/questions:

1) Why did you decide to give less weight to the number of days with a tornado of path length .ge. 15 mi. than to the number of days with a tornado path length .ge. 4 mi.? Seems sensible to give the former more weight since chasers would probably consider a season to be better if they got to chase a bunch of long-tracked tornadoes rather than medium/short-tracked tornadoes.

Longer-track tornadoes are actually double or triple counted in the other categories, so that's not the case; a 20-mile track is going to be counted in every tornado category (all, >=4, >=15). Sorry, I don't think I explained that aspect very well, and it's an incredibly crude way of weighting things. The 5-10-5 weighting for tornado counts (all->4->15) means >4-mile tracks are weighted 3x as much as short tracks, while >15-mile tracks are weighted 4x as much.

Perhaps the best way of scoring is to base everything on individual days, but this was something I whipped together quickly in Python without wanting to go too far down the rabbit hole (for now). I can see the merits of using a more nuanced formula for the number and track-length of tornadoes on each individual day, and then just summing all the single-day scores for a month/season/year. That would eliminate this awkward separation of "total count" and "day count" metrics. The discrete path-length categories could also be eliminated. A logarithmic scoring function of path length, for example, would accomplish what I'm trying to do here in a cleaner way: reward longer tracks, but mostly within the 0-15 mile range and only a little beyond that.

2) Have you considered performing the linear regression based on tornado counts with (estimated) ratings of (E)F0 and (E)F1 rather than final relative score? I seem to recall that the majority of the increased counts of yearly tornadoes over the years has been due to more low-end tornadoes spotted/reported. I suppose, however, your current method also kind of takes into account the increased availability of chasing since the 1950s. I'm guessing it wasn't even possible for as many people to get out and traverse the central US searching for tornadoes back then with the lesser technology (or maybe the technology was fine and the reasons were more societal and I'm just wrong).

Yeah, that's where most of the inflation comes from. Actually you could make the case that significant tornado numbers may have been inflated somewhat in the past compared to today's standards.

Thanks for pointing this out. When I have time, I'll see if I can find a few relevant papers on report inflation, which I neglected to do for this first attempt. I'm sure there's a better way to handle inflation that could be adapted from existing research.