Fraud Allegations for the Center for Severe Weather Research (the DOWs)

[Alex Elmore]

It is very much a tangential field when compared to tornado warnings and staplers are not a good analogy to the problem. As far as I know, no one is using machine learning to attempt to issue real-time tornado warnings because that field is simply too complex.

As discussed previously in other threads that you've been involved in, the CIMSS ProbSevere tool has the product "ProbTor", an AI/ML-based tool used to diagnose the tornado potential of a storm. While it is not used solely to determine if a warning should be issued or not, it serves as a confidence builder/reducer. Radar still reigns supreme, though.

While the application of AI differs in the scenarios presented in this discussion, the methods in which AI is used to identify something is similar across fields: you gather training data (i.e., images of staplers or radar signatures), have the AI/ML study it, and then assuming the training data was properly diverse, the AI/ML can properly identify a stapler, or hopefully, a tornadic signature. Which leads to the next point...

My comment about DOW radars is correct. If they were making a material, positive contribution, warning quality would be going up. Instead it is going down. I do not blame that on the DOWs. I am saying they are irrelevant and I see no sign of that changing. Rob has posted a number of items attempting to defend the DOWs but none of those items is pertinent to real-time tornado warnings and tornado forecasts. I note no one else has attempted to refute my observation about the DOWs with evidence. That is not surprising to me because I do not believe that evidence exists.

Putting it plainly, seeing as the NWS is the sole provider of public tornado warnings, and the NWS uses WSR-88D dual pol data (and maybe TDWR data if they're fortunate), then any research that is done on tornadoes that doesn't focus specifically on that data is a waste of time. You could add on GOES and lightning data, but the point still stands. Tornado research is a long game with currently very little near-term payoff in terms of directly impacting warning operations. So yes, putting it that way, the DOWs are useless. Orf and team's research is useless. However, in the grand scheme of things, it is not. Each of these field campaigns each of the high-resolution model runs chips away at building that diverse training data that may eventually be used to feed an AI/ML-based tool. They offer insight into what is going on near or at the surface prior to and during tornadogenesis, such as near-surface formation of the El Reno 2013 tornado discussed by Bluestein et al. 2019 or vorticity currents found in Orf et al.'s work. This sort of work has been used by various NWS groups, such as the Tornado Warning Improvement Project, to diagnose radar signature precursors to tornadogenesis. At the very least, it highlights that the forecast enterprise could benefit from the high-resolution data the DOWs gather in real time, maybe making the push to have an operational phased array radar network or something of the like before I die.

There have been many scientific advancements throughout history that have come about from research that was indirectly related. While that line of thinking can not be used to justify every single bit of research out there, saying that the DOWs have no impact whatsoever on the warning paradigm is clearly hyperbolic and narrow minded.

 

[Mike Smith]

Rob,

I completely agree with you that tornado warnings should be more accurate than they were 20 years ago. Unfortunately, they are not.

NWS lead time has been cut in half and is about at 1995 levels. CSI is about at 2005 levels. I'm not the only person who has noticed.
https://www.washingtonpost.com/news...es-as-researchers-struggle-to-understand-why/

We had an unwarned fatal tornado yesterday evening, tragically. Mobile Homes MUST Be Evacuated Ahead of Tornado Warnings! As you'll see at my blog post, this was a pretty obvious situation, even though it was QLCS.

The money that has been going to DOWs and field programs needs to be immediately diverted to figure out why we are experiencing these problems with the tornado warning system.

Mike

 

[Noah Anderson]

The money that has been going to DOWs and field programs needs to be immediately diverted to figure out why we are experiencing these problems with the tornado warning system.

Why is this a zero sum game? How about we fund both?

 

[Mike Smith]

Alex, Noah and Everyone,

Please allow me to answer your comments.

Again, congratulations to STL NWS, Alex, on a great job Sunday night. Really well done.

The situation in which we find ourselves is nothing like the Apollo program. By 1967, Apollo hadn't even begun. The space program was on a tremendous upward trajectory (the tragedy of Apollo 1 notwithstanding) with Mercury and Gemini. It certainly was not deteriorating. In 1944, after only two years of work, the Manhattan Project had atomic bombs that worked.

The NWS tornado warning program peaked in accuracy in 2005-2010 and has been deteriorating since including the horrific tragedy of the Joplin Tornado. JLN was caused by the terrible NWS warning quality that day, the misapplication of tornado sirens by local emergency management, and the fact the tornado was rain-wrapped and invisible to those along its path. The combination lead to the loss of 161 lives. See: Amazon.com: "When the Sirens Were Silent" How the Warning System Failed a Community eBook : Smith, Mike: Kindle Store

I am hardly the only person who has noticed the tornado warning issues. Here's a 2007 article from the Washington Post.
https://www.washingtonpost.com/news...es-as-researchers-struggle-to-understand-why/ KWTV in OKC six months ago was running a promo that said, "We don't wait for the National Weather Service."

Alex wrote, the CIMSS ProbSevere tool has the product "ProbTor", an AI/ML-based tool used to diagnose the tornado potential of a storm. While it is not used solely to determine if a warning should be issued or not, it serves as a confidence builder/reducer. Radar still reigns supreme, though. While I don't know how long ProbTor has been in the field, it does not seem to be working, based on the NWS's own statistics. How long a period was ProbTor independently tested in a real-time warning environment before it was put in the field?

The NWS certainly is not the only source of tornado warnings. WeatherData/AccuWeather issues its own tornado warnings for its clientele. In view of issues with the NWS warnings, more and more television stations are issuing their own warnings. You might have seen the VP of DTN on LinkedIn last week bragging about their tornado warnings and I know of five other commercial companies that are now doing the same. I do not view this as a positive development. But, as long as NWS tornado warnings continue to deteriorate, the private sector will step in to fill the gap. Saying we should continue to spend money on DOW observations is like saying we should improve the quality of the popcorn while the movie theatre is on fire.

It isn't like we started doing with DOWs 2 or 5 years ago. We have been chasing tornadoes with DOWs every year for more than a quarter century! See: A Mobile Mesonet for Finescale Meteorological Observations

We have done six VORTEX's plus other field programs about and related related to tornadoes (PECAN, etc.). Most of that data is still sitting on the shelf (I directly asked one of the PA's). I participated in the NSSL forecast testbed the summer of 2016 where we evaluated a number of satellite/statistics and other datasets. DOWs were never mentioned in the classroom discussions when the instructors explained how the statistics and techniques were evaluating were derived.

From 1967 to 1971 NSSL ran a super-mesonetwork (stations as close as every five miles) in central Oklahoma in a square from around OKC to Chickasha. Not only did they have 100+ surface stations, they launched rawinsondes at 3-hour intervals when storms were expected and at 1-hr intervals once storms had fired. They did that OKC, LTS, END, CHK, FSI and CSM. That huge data set produced just a single paper -- on trends in the lifted index. That's it. A huge investment in resources practically wasted. That data would be incredibly valuable today to use for mesoscale modeling experiments. Unfortunately, all of it burned in a fire at an NSSL storage unit about 15 years ago. This demonstrates that presuming the data will be used at some point in the future may not turn out to be the case.

As to using the DOW data to train AI, etc., what is wrong with that dataset as it currently exists? How many of the DOW in-situ observations have actually been used? Why do the taxpayers have to continue to fund more?

Noah, you ask the good question, "Why not fund both?" And, that may seem reasonable when the Administration throws around money by the trillions. But, as a taxpayer, I want the meteorological community to be good stewards of tax dollars. If someone (Bill Gates, Jeff Bezos) wanted to fund DOWs and future VORTEX-type experiments I would be 100% for it and would write him or her a thank you letter. But, given how little we've gained, I would say that we need to wring every bit of data we can from existing datasets before we ask for more, especially when our nation is trillions of dollars in debt.

Want to improve tornado warnings? Fund gap-filler radars. Train meteorologists in simulators in a step-by-step process the way may generation was trained (I believe a lot of the issue with NWS TORs is the experience that has walked out the door as my generation of mets has retired). That is the way to address the immediate problem, not DOWs or VORTEX and hoping that data eventually proves useful.

About 40 years ago, medicine learned if it was going to advance, it was going to have to stop sweeping mistakes under the rug. Most all hospitals now have weekly mortality and morbidity (M&M) meetings where everyone candidly discusses their mistakes. Meteorology simply refuses to do this. Heck, the NWS doesn't even do its weak "service assessments" any more when the quality of its performance is in doubt!

That is why I a strongly advocate an independent National Disaster Review Board (NDRB), modeled after the National Transportation Safety Board, that would take a hard look at these issues from all sides and advise NWS and private sector entities how to improve. An excellent candidate for this type of investigation would be the recent flash flood in Tennessee that killed 18: The Catastrophic Tennessee Flash Flood - Why The United States Needs a National Disaster Review Board and, the multiple unwarned tornadoes of this past summer (two examples here: Bensalem Tornado: Another Dangerous National Weather Service Warning Miss - and - here: Friday: Another Dangerous Missed Tornado Warning ).

It is entirely possible an independent NDRB might say I am wrong about all of this and that's fine. But, right now, we have entities -- at best -- examining themselves without a fresh perspective. The NTSB is one of the most successful things the federal government has ever done. Let's replicate that success!

Finally, none of what I have written is personal. Terms like "narrow-minded," etc., don't get us anywhere. If meteorology is ever going to mature as a science, we need to stop with the name calling and be open-minded to the wide perspective of views from throughout our field.

Thank you for your kind attention and your consideration of my perspectives.

Mike

 

[Mike Smith]

Good point - here’s a link to his research. It’s eye opening!

Leigh Orf | Simulation and visualization of thunderstorms, tornadoes, and downbursts

While I may have missed something, I perused Lee Orf's page back to April, 2019, and looked for anything about the DOWs. I don't see it. Would direct me to it with a link? Thanks!

 

[rdale]

The BAMS article from 2017 has multiple references to field observations.

Here's a similar page he hosts for those who find his primary resources to be a little too hard to understand:

Education | Leigh Orf

orf.media

 

[Mike Smith]

Everyone,

My post #29 above is incorrect as to the location of the unwarned fatal tornado. It occurred in a vehicle at Moss Point, MS. There was no tornado nor severe thunderstorm warning for that storm.

The mobile home that was obliterated at Picayune -- while also unwarned -- was not the location of the fatality. The radar data presented for that storm is correct.

As it turned out, there were at least two unwarned, damaging, tornadoes in southern Mississippi Wednesday evening.

I apologize for the error. Unfortunately, the Stormtrack software will not allow me to correct it, so I am putting this in bold so, hopefully, everyone will notice.

Thanks to John Robinson for pointing it out.

Mike

 

[Mike Smith]

The BAMS article from 2017 has multiple references to field observations.

Education | Leigh Orf

orf.media

Rob, thanks so much for making my point conclusively. If Orf (2017), can achieve his impressive results with a data from a field project from the 1990's and DOW data no more recent that three years before the paper was published then subsequent field projects and DOW intercepts are irrelevant to this research. In other words, we seem to be getting little additional in return for the millions of dollars we are spending on this research.

Thanks, Rob.

 

[Alex Elmore]

Alex wrote, the CIMSS ProbSevere tool has the product "ProbTor", an AI/ML-based tool used to diagnose the tornado potential of a storm. While it is not used solely to determine if a warning should be issued or not, it serves as a confidence builder/reducer. Radar still reigns supreme, though. While I don't know how long ProbTor has been in the field, it does not seem to be working, based on the NWS's own statistics. How long a period was ProbTor independently tested in a real-time warning environment before it was put in the field?

Correlating one tool/guidance source to warning performance will yield inaccurate results, as it assumes that the tool/guidance source is used uniformly across all NWS offices by all warning meteorologists in all tornado warning situations, which is not the case. It, along with many other types of tools/guidance sources, is not meant to be a silver bullet and used on its own, which provides addition nuances into drawing connections.

Want to improve tornado warnings? Fund gap-filler radars. Train meteorologists in simulators in a step-by-step process the way may generation was trained (I believe a lot of the issue with NWS TORs is the experience that has walked out the door as my generation of mets has retired). That is the way to address the immediate problem, not DOWs or VORTEX and hoping that data eventually proves useful.

NWS meteorologists, as brought up in prior discussions, are trained in simulators in a step-by-step process. This training is called the Radar Applications Course (RAC) and every NWS meteorologist is required to go through it before they can reach the warning desk. Each NWS office also has a Weather Event Simulator (WES) in which the SOO can walk through any type of weather event, including radar cases, with meteorologists to train them or maintain proficiency. As also has been discussed in prior conversations, maintaining experience within an NWS office is certainly a topic of concern, but assuming that experience will sharply drop off with the retiring of an older generation assumes that the younger generations, some of which have still been working for 20+ years, are somehow less intelligent/capable/skilled than their predecessors. That also assumes that the data available has stayed the same over the last several decades, which is far from true (i.e., dual pol, recent GOES updates, mesoanalysis data, hi-res radar data, and hell, even the introduction of the WSR-88D itself). Neither of these assumptions are true, thus, the idea that a drop-off in performance is directly and solely tied to retirement is misleading and fails to address the nuance of the situation. Lastly, lets not forget that the average annual number of tornadoes artificially increased over several recent decades due to enhanced public awareness as well as advancements in technology, translating to that there were roughly a couple hundred tornadoes a year that were going undetected by meteorologists of yesteryear. This point is absolutely not made as a "see, your generation sucked too!", but to put those fond memories of the "golden years" in perspective.

As to using the DOW data to train AI, etc., what is wrong with that dataset as it currently exists? How many of the DOW in-situ observations have actually been used? Why do the taxpayers have to continue to fund more?

Back to the matter at hand, there is nothing necessarily wrong with the data that is currently unavailable for training AI. Using DOW data for AI training was simple example for how eventually something may be used. I have no idea if this would ever be done. However, the point was made to address the idea that the DOW data makes no contribution to the warning process and its inaccuracies. It's also not meant to say that the cost of funding DOW-based campaigns is cost-effective. It very well may not be, but that still doesn't mean the data is not useful now or ever.

Finally, none of what I have written is personal. Terms like "narrow-minded," etc., don't get us anywhere. If meteorology is ever going to mature as a science, we need to stop with the name calling and be open-minded to the wide perspective of views from throughout our field.

Neither is anything I said. I can label a thought as "narrow-minded", not the person, which is what I did here. I do generally agree that the advancement of any area of science comes from new and sometimes "against the grain" types of ideas; however, that doesn't mean all such ideas are acceptable or worth pursuing. That's why the scientific process and peer-reviewing (what it's meant to be, not how it is sometimes practiced) is a beautiful thing.

 

[Mike Smith]

Research papers are not written for people like you - but for scientists. We use this research to help improve the process, and in turn save more lives.

Rather than concede my point, my critics cannot stop themselves from making this personal.

Since you believe I am not a scientist, Rob, let me list the evidence and those reading this can decide for themselves.

• "Special Award for Lifetime Achievement," National Weather Association. The citation refers to, among other things, my work in the field of tornado science and storm warnings.
• Fellow of the American Meteorological Society (AMS)
• Certified Consulting Meteorologist
• AMS Award for Outstanding Contribution to Applied Meteorology. This citation refers to my work in tornado warnings and tornado research.
• AMS Award for Achievement in Corporate Meteorology for my training course for university meteorology instructors.
• Certified expert in the federal and state courts pertaining to tornadoes, thunderstorms and aviation.
• Peer-reviewed papers published on the topic of downbursts and aviation hazards
• Per Dr. Ted Fujita, my series of seven photographs in July, 1978 confirmed the existence of downbursts which had been a topic of fierce scientific debate.
• Engineering Innovation & Showcase Award for StormHawk®, the first-ever GPS storm warning and reporting device, Consumer Electronics Society
• Author of numerous articles in the popular press (Slate, NYT, WaPo, etc.) and in the meteorological press (e.g., Weatherwise, etc.)
• One book on the topic of tornadoes, hurricanes and downbursts that is used as a textbook at several universities' meteorology classes
• One book on what went wrong during the Joplin Tornado. It is a forensic analysis written in a style for the public.
• With Dr. Joe Golden, chair the American Meteorological Society's conference on downbursts in the wake of Delta 191. Was member of the AMS Committee on Aviation Meteorology that assembled, with Boeing and the FAA, the first pilot training course for downburst avoidance
• Presented papers at numerous AMS and NWA conferences
• 30+ U.S. and foreign patents in the field of applied meteorology, GIS/GPS, emergency management, and search & rescue
• In ten days, November 8, by invitation, teaching a course at the University of Oklahoma

There truly is much more, but this should be enough to allow readers to determine whether I am a "scientist" regardless of Mr. Dale's comments.

 

[Mike Smith]

It, along with many other types of tools/guidance sources, is not meant to be a silver bullet and used on its own, which provides addition nuances into drawing connections.

NWS meteorologists, as brought up in prior discussions, are trained in simulators in a step-by-step process. This training is called the Radar Applications Course (RAC) and every NWS meteorologist is required to go through it before they can reach the warning desk.

...The idea that a drop-off in performance is directly and solely tied to retirement is misleading and fails to address the nuance of the situation. Lastly, lets not forget that the average annual number of tornadoes artificially increased over several recent decades due to enhanced public awareness as well as advancements in technology, translating to that there were roughly a couple hundred tornadoes a year that were going undetected by meteorologists of yesteryear. This point is absolutely not made as a "see, your generation sucked too!", but to put those fond memories of the "golden years" in perspective.

Back to the matter at hand, there is nothing necessarily wrong with the data that is currently unavailable for training AI. Using DOW data for AI training was simple example for how eventually something may be used. I have no idea if this would ever be done. However, the point was made to address the idea that the DOW data makes no contribution to the warning process and its inaccuracies. It's also not meant to say that the cost of funding DOW-based campaigns is cost-effective. It very well may not be, but that still doesn't mean the data is not useful now or ever.

Neither is anything I said. I can label a thought as "narrow-minded", not the person, which is what I did here. I do generally agree that the advancement of any area of science comes from new and sometimes "against the grain" types of ideas; however, that doesn't mean all such ideas are acceptable or worth pursuing. That's why the scientific process and peer-reviewing (what it's meant to be, not how it is sometimes practiced) is a beautiful thing.

Alex, thank you for taking time for a comprehensive reply. I have bolded where I think our biggest disconnect lies.

• Multiple NWS meteorologists have told me, face-to-face, they believe their training in tornado warnings is inadequate. I don't know where the truth lies. This is during the last two years.
• It may be that, in the last decade, we have made the tornado warning process too complicated. We may have meteorologists trying to analyze the guidance rather than the data. The former can be counterproductive.

As to your other comments: I've never said that loss of experience is the reason. I've offered it as a hypothesis along with inadequate training and a number of other ideas. That is why we need a National Disaster Review Board.

You are conceding that the DOW data hasn't materially been used for tornado warnings (and, I'll add, forecasting). I don't know why it has taken this long to get to this point. You agree it may not be cost-effective. Conclusion: We don't need to, in the immediate future, continue to fund this expensive data acquisition. If the on-the-shelf data can be used for better warnings in the future, great! We can restart the process if necessary.

Thank you for your last paragraph.

 

[Jeff House]

Mike Smith has owned this discussion hands down. Alex may be co-owner but the usual suspect rdale trolling has once again proven himself unprofessional and unbecoming of the field.

Twitter nagging the NWS gets annoying, but once in a while it's right. DDC No.

Let's go back to when I was actually a fairly green Met. I'm sitting on the WeatherData (eventually AccuWx Enterprise Solutions) warnings desk. Severe wx is in progress in Kansas. I see a textbook tornado signature on radar. Cell has a history; so, it's not a hard decision. I give our B2B clients a tornado warning. Hoisington, KS has a tornado about 10 minutes later. Quality lead-time.

Reality is that some obvious warnings are missed. If the commercial airlines are held accountable by the NTSB, something as critical as weather warnings should have independent oversight. That's said with so much respect for my colleagues and friends in the NWS.

I believe the FAR infatuation is hurting POD stats. Public knows a tornado warning means take cover, not necessarily it's gonna hit my house. Warning fatigue is real, but not as big a problem as getting dead. Better give a couple points on the FAR to improve POD.

 

[Mike Smith]

Thank you for your comments, Jeff.

I want to enlarge on Hoisington a bit. The tornado was a classic signature of EF-4 intensity. One person died. WeatherData got it correct as did every Wichita TV station. The NWS did not issue a TOR until the tornado was literally halfway across the town.

 

[rdale]

I believe the FAR infatuation is hurting POD stats. Public knows a tornado warning means take cover, not necessarily it's gonna hit my house. Warning fatigue is real, but not as big a problem as getting dead. Better give a couple points on the FAR to improve POD.

I don't think Mike's issue is his infatuation with FAR - but his inability to see that others actually know some (if not more) about tornadoes than he does. Let's not even get into the EM discussion He absolutely is unprofessional when discussing this topic, but as long as he gets in a link to his book it's okay.

Back to your topic - yes, FAR gets overblown by social mediarology experts. Many counties get one tornado warning every few years. I've never heard anyone say "Well three years ago there was a warning for my county, and there was no tornado, so obviously I won't take shelter for this one." It's pretty rare that EF3-5s are missed these days, and that's due in large part to VORTEX and DOW and the like.

 

[Mark Blue]

I kind of hope this discussion has run its course lol