On the Use of Indices and Parameters in Forecasting

[Darren Addy]

While most of us are impatiently twiddling our thumbs, I thought it might be a good time to focus some attention on a paper that I can't stop thinking about by Doswell & Schultz entitled: "On the Use of Indices and Parameters in Forecasting".

If you are not a meteorology student, don't let the formulas throw you. It is worth a read even without the details.

I think Doswell makes some valid points, but in other ways I think it is also a paper that could use a lot of revision. The body of work really doesn't fully accomplish what the introduction says it will do, and the Conclusion is almost completely unrelated to the goals set forth in the introduction. Nevertheless, I think it is worth considering what he has to say. (Also the peer reviewers comments/discussion at the end of the paper are educational.)

The basis of Doswell/Schultz's argument is pretty simple and irrefutable: Any time you create an index or composite the accuracy of the output is going to be affected by the accuracy of the individual component numbers. So looking at only the resulting numbers, without understanding how they might change if one of the components doesn't match expectations can lead to errors. Ultimately he argues against over-reliance on these products and the inherent superiority of a human forecaster's skill.

We're all probably familiar with the old "garbage-in/garbage-out" principle, and understand that things like model skew-Ts can be little more than "best guess" fiction, particularly as the timelines stretch out into the future.

Even though the basis for their argument is essentially irrefutable, it doesn't mean the conclusions that they reach or the arguements and examples they use are necessarily the best. Curious as to what others thoughts might be.

Like anything, meteorological formulas and models are constantly under review, revision and will probably better reflect realities as time goes on, but if anything Doswell/Schultz's paper impressed me with the importance of understanding the individual components (for example, that go into producing a CAPE number) rather than simply looking at the CAPE number itself.

I'd appreciate hearing anyone elses thoughts on the subject, particularly after you get a chance to digest the paper:
http://www.ejssm.org/ojs/index.php/ejssm/article/view/11/12

This is a related article from Doswell: What's wrong with Indices and Parameters?

 

[Mike Johnston]

Although I'm in no position to dispute the general assertions of the technical paper, I think the tone of the related essay is unnecessarily dismissive of the value of the "diagnostic variables", as Doswell calls them.

To quote, he says: "For them to issue a product is to endorse its contents and encourage the use of any products contained on the page." He is referring to the SPC's mesoscale analysis page.

For one, I don't view the mesoscale analysis page as a "product" of the SPC. The products are their forecast products - outlooks, watches, etc. The mesocale analysis is essentially a data portal that is oriented to convective parameters, and is appropriately labeled under "research tools" on the SPC site. How, or what, information from this analysis to incorporate into a forecast (be that an individual chaser's forecast or a WFO's forecast) is up to the forecaster, obviously varies from situation to situation, and I don't believe carries any "endorsement" from the SPC.

Also, on the use of composite parameters like the EHI, Doswell correctly points out that the parameters' formulas aren't necessarily rooted in physical relationships. True, but I've never seen where the developers of the EHI made any such claims and I think most knowledgable users (even us "amatuer" forecasters) understand this.

Bottom line - would you rather make a forecast with or without access to these diagnostic variables available on an hourly basis, such as presented on the mesoanalysis page?

 

[Darren Addy]

Good points, Mike. A couple of other thoughts I had...

In speaking about "volatile parameters" he states "Using such a variable as a forecast of weather to come is a much riskier venture". I think this is an area of relative risk. To assume no risk, one would not make a forecast at all. But that would increase risks for the people living in a forecast area. So making some assumptions and taking some risks is worthwhile and even necessary for progress to be made in any field (I would argue).

Secondly, I think that the indices and composites are in a way a response to a form of information overload. We need to feed the models and we need to distill and combine real time observations. He admits that "deduced variable" like mixing ratio and potential temperature are more useful than their component raw observations for example.

Lastly, his conclusions always seem to come back to a defense of the human over the machine. This, to me, is an emotional argument that contaminates an otherwise clear-eyed scientific look at the subject at hand. It reminds me of the old days when chess players held onto a belief that the machine would never "play" better than a human Grandmaster. It is almost as if he is arguing that any meteorologist that works in the area of developing indicies, composites, or model algorithms is not acting in the interests of the human's job markets. This is :
a) not consistent with the sciences' pure pursuit of knowledge
b) not logical, since it could (perhaps) result in a need for fewer "forecasters" but more jobs on the data/I.T. side. I think he is being a bit of a romantic defender of weather forecasting as a human profession. For the record, I think it is a long way from going extinct... but I think it compromises the reasons for the other conclusions that he is reaching for.
c) not reasonable to conclude that humans will be completely replaced at all.

Even a pilotless drone in the Air Force requires a real human back at the controls to make real time decisions based upon what it sees/reports.

 

[Derrick Herndon]

I lost some respect for Doswell while reading this. I too agree that the emotional tone and "dumbing" down of forecasters hurts his argument. So lets see:

1) Yes the observation network lacks sufficient spatial resolution to adequately sample the atmosphere. We knew that.
2) The atmospheric variables that lead to supercell development may not co-exist at the time of the observations. Uh yeah we knew that too.
3) No single index is going to forecast all tornado events with zero error. Yep knew that too. Moreover I take issue with the statement that the derived parameters that make up the indices are not adequate predictors of tornadoes themselves therefore the index is of no value. Well yes CAPE alone is not a strong predictor of tornadoes but large CAPE along with large low level shear gets my attention. So to me a parameter that combines those 2 values makes sense to me even though I will ALWAYS look at the individual constituents anyway in order to get an idea of their spatial scale and amount of overlap.
4) I am not sure why Doswell chose to focus on analysis only. Who decides to chase based only on the morning sounding at a single location? A better argument would have been to discuss the potential errors in the indices as a function of model error. Particularly data assimilation.
5) Which brings me to my next point: soundings are no longer the only source of upper level data and have not been for years. More and more data assimilation of satellite observations is filling in the gaps between the upper air stations. No mention is made of this nor the benefits/downsides of this data source.
6) Also the issue of time lag has me a little baffled as well. Why not quantify that statement with some data showing this lag if there is one? Seems that would be of some value.
7) I understand the fear involved in this paper. The fear that forecasting is a dying art. That computers are replacing experience and good ole fashioned intuition and understanding. The problem though is you can not manufacture experience and plug it into a forecast office. But you can build a better model or help young forecasters use their time more efficiently. That is one of the motivations of these parameters. The idea that the parameters are being used in a vacuum without understanding of how they are derived I think is false. If it is true then I would like to see that documentation. Was an event missed because of it?

The paper does seem to be tailored not to the professional meteorologist but students who believe all they need is a single index value to forecast severe weather. But as one who was a meteorologist student I would say that at least at Florida State my teachers did a pretty good job of making that point.

 

[Robert Edmonds]

I agree with most of the comments here that Doswell said allot of what is already known. Even not being a trained meteorologist, I think a person spending just a little time trying to forecast or develop their target for the day will soon realize that just using a single parameter blindly is certainly not a wise idea. I'm not quite sure of the purpose for such a paper, like Derrick said, is this more intended for students? On a side note, I respect allot of the research he's done, I think I even cited a paper or two of his durring a project, but he seems to be more focused on the negative aspects of things.