April 27, 2011 outbreak - NWS service assessment

[Jeff Duda]

http://www.nws.noaa.gov/om/assessments/pdfs/historic_tornadoes.pdf

I am particularly interested in findings 9 and 10 on pages 18-19 (26-27 of the PDF) regarding the content of the text messages in tornado warnings:

Finding 9: Forecasters had differing interpretations of the instructions in NWS Directive 10-511
concerning “Tornado Emergency.” This inconsistency caused confusion among those media
partners whose market areas were covered by multiple WFOs.
Recommendation 9: NWS regional headquarters should ensure that WFOs in their areas of
responsibility consistently apply instructions for the application and use of “Tornado
Emergency.” NWS regional headquarters should inform OCWWS of any requirements to clarify
or provide more detail on the use of “Tornado Emergency” in NWS Directive 10-511.

-WFOs BMX and FFC used the wording “Doppler radar indicated” for the basis of their
warnings and statements; however, staff were aware that large destructive tornadoes were on the
ground based on spotter reports, debris ball signatures on radar, or through visual confirmation
via television (see Figure 8). The assessment team echoes the findings of the Super Tuesday
Tornado Outbreak of February 5-6, 2008, Service Assessment: Usage of generic wording may
lead people in the warned area to feel the threat is not as great as it actually is.

Finding 10: Several WFOs used the generic “Doppler radar indicated” wording in their tornado
warnings despite evidence of a tornado in progress.
Recommendation 10: When WFOs have ground truth reports, they should refer to them in
severe weather products, rather than “Doppler radar indicated.” OCWWS and the regions should
review NWS Directive 10-511 and ensure the directive supports this recommendation.

It seems to me that the usage of "tornado emergency" has gotten out of hand, as no one person or office can seem to agree on when and how to use the term. I used to support its use, but I now think the term should be replaced since I don't think the phrase "tornado emergency" really helps anyone, especially given how it is supposed to be used according to NWS Directive 10-511:

In exceedingly rare situations, when a severe threat to human life and catastrophic damage from a tornado is imminent or ongoing, the forecaster may use the terminology "TORNADO EMERGENCY FOR [GEOGRAPHIC AREA]" in the third bullet of the warning. Additionally, in such a situation, this terminology should only be used when reliable sources confirm a tornado, or there is clear radar evidence of the existence of a damaging tornado such as the observation of debris.

The THIRD bullet??? I think if the wording was that important it should be the first bullet. IF the NWS were to continue going down this road, then instead of issuing a tornado warning and calling it a tornado emergency (which, of course, does not confuse anybody[/sarcasm]), NWS meteorologists should issue a tornado emergency, a strictly different product than a tornado warning.

However, like I said, I think the phrase "tornado emergency" should instead be replaced by very strong wording in the warning. I think some of the issues with getting people to heed warnings is how they sound to someone hearing them. I realize almost any governmental organization is going to be more conservative so as to avoid causing panic and what not, but in situations like this where the environment is dangerous and damaging/deadly tornadoes have already been spotted or can be seen on TV or chaser streams, saying

"this is a life threatening situation"

at the bottom of the warning text isn't good enough. First of all, the NWS warning system is antiquated and needs an overhaul. Second, for situations like this, the wording needs to be more like

"if you aren't in a secure shelter, such as an underground storm cellar, you are not likely to survive a direct impact from this tornado"
or, even more extreme,
"this tornado will kill you if you do not take adequate shelter"

would probably get peoples' attention better and cause more to heed the warning (so long as its use is not abused...unless the warning overhaul includes that mention in a standard tornado warning). I don't think the excessive loss of life was entirely due to NWS warning policies, however. There certainly was a unique and unfortunate confluence of failures/difficulties that contributed to the death toll (i.e., NWR going out, KHTX going down etc.).

Switching gears: As far as SPC is concerned, I think high-end situations like this call for adjustments to the probabilities associated with convective outlooks. For example, the 2000 UTC SWODY1 had only a small area of 45% probability for tornadoes. Looking at the observed reflectivity at around the time that convective outlook was issued, and given the volatile environment, I don't think a >45% probability for tornadoes, with only a slight amount of area added outside that region of 30%-45%, really gives a true sense of the likelihood of tornadoes in that area, especially given that the most lethal tornado was within 30 minutes of touching down by the time the outlook was issued. I understand that part of those probabilities includes forecaster certainty, but I don't see how a(the) forecaster issuing that outlook can only see the chance being "greater than 45%". That may be true, but I think the probability was way over 45%. Hell, it should've been nearly 100% for tornadoes. I don't see why a 60% or 75% or even a 90% contour can't be added in situations like this. While it may be true that areas within the 45% contour technically have a greater than 45% chance of seeing a tornado nearby, I think that undercuts the true potential. Most people aren't going to see that and think "well, it's only a 45% chance...".

Anyway, I'd love to see some discussion sparked about this (and other findings/issues brought up in the service assessment).

 

[Patrick Marsh]

Regarding the phrase "Tornado Emergency" being in the third bullet, the short answer is it has to be. On 27 April 2011, NWS Huntsville DID put the phrase in the first bullet...and within seconds of that going out the Severe Weather Services Coordinator for the NWS had his phone go into melt down. Why? Many of the warning vendors are expecting certain phrasing to be in certain places for parsing purposes. When you change that, warnings don't get parsed. This is why the recommendation was to put the text in the third bullet -- which isn't used for parsing.

 

[Patrick Marsh]

Switching gears: As far as SPC is concerned, I think high-end situations like this call for adjustments to the probabilities associated with convective outlooks. For example, the 2000 UTC SWODY1 had only a small area of 45% probability for tornadoes. Looking at the observed reflectivity at around the time that convective outlook was issued, and given the volatile environment, I don't think a >45% probability for tornadoes, with only a slight amount of area added outside that region of 30%-45%, really gives a true sense of the likelihood of tornadoes in that area, especially given that the most lethal tornado was within 30 minutes of touching down by the time the outlook was issued. I understand that part of those probabilities includes forecaster certainty, but I don't see how a(the) forecaster issuing that outlook can only see the chance being "greater than 45%". That may be true, but I think the probability was way over 45%. Hell, it should've been nearly 100% for tornadoes. I don't see why a 60% or 75% or even a 90% contour can't be added in situations like this. While it may be true that areas within the 45% contour technically have a greater than 45% chance of seeing a tornado nearby, I think that undercuts the true potential. Most people aren't going to see that and think "well, it's only a 45% chance...".

Just for clarification, the SPC *can* issue a 60% contour (http://www.spc.noaa.gov/misc/SPC_probotlk_info.html). However, keep in mind that the probability is for a tornado to occur within 40 km of a given point. Even with all of the tornadoes, we probably didn't exceed the 60% threshold for a lot of the 45% contour. Could there have been a 60% contour? Yeah, definitely up in northeast Alabama. However, at 2000 UTC there was no way of knowing NE AL was going to see the highest concentration.

You do raise an interesting point though about people responding to probabilities. 45% is certainly "low" in a probabilistic sense, but not in a tornado sense. I personally think the better question to be asking is, "What is a better way to convey the threat than probabilities?" Should we relate the threat to a normal baseline? (forecast prob / climo prob) This would allow someone to say, "There is XXX times greater potential for having a tornado today than normal?" I'd suggest we need to have some sort of combination of the two...

 

[Jeff Duda]

Regarding the phrase "Tornado Emergency" being in the third bullet, the short answer is it has to be. On 27 April 2011, NWS Huntsville DID put the phrase in the first bullet...and within seconds of that going out the Severe Weather Services Coordinator for the NWS had his phone go into melt down. Why? Many of the warning vendors are expecting certain phrasing to be in certain places for parsing purposes. When you change that, warnings don't get parsed. This is why the recommendation was to put the text in the third bullet -- which isn't used for parsing.

Thanks for the inside information, Patrick. However, I think that just proves my point that the warning system - especially the text part - needs to be redone. This is along the same lines as WFOs issuing county-based warnings in the storm-based warnings era simply because they don't want to get an earful from an EM who can't or doesn't want to only warn parts of his county instead of all of it.

As far as the SPC part goes, Patrick, since you are highly skilled with Python and whatnot, would you be willing to put together a map showing the tornado tracks with a 25 mi buffer around them so we can get a sense of the spatial distribution of the tornado tracks from that day? I still think a higher number was warranted to give a better sense of the threat, but perhaps you are right about the true density of tornadoes not fulfilling the need for a 60%.

 

[Simon Andersen]

"All tornado fatalities occurred within the boundaries of tornado watches and were preceded
by tornado warnings." Maybe they should stop issuing watches and warnings? I found the discussion regarding barriers (in language, administration etc.) interesting. It's easy to ridicule the disconnect in tornado mapping between administrational boundaries but really, it must be staggering to have to piece together so much information. It's commendable that you do what you can to inform new citizens who don't speak english, although I'd make sure I was on top of everything if I moved to another country where they spoke a completely different language. I noted the mention of NWS-regions, are they a different leve from offices? I look forward to reading the whole assessment, thanks for notifying of it.

 

[rdale]

Regions set policies for local offices and they may differ from region to region. Not sure why, but it does lead to concerns (especially when one region, who I won't name but exists in the south, doesn't put taglines on warnings so that computers can read them in.)

 

[Mike Johnston]

Re: the probabilities question, perhaps it would make sense to use Dr. Forbes' parameter for his TorCon index which is based upon within 50 miles of a point. Seems like he thought through this very well, as it roughly calibrates to a 1-10 (10% to 100%) scale. This kind of scale is more easily understandable by the average consumer. Of course, the average consumer probably doesn't even know what an SPC Outlook is, let alone reads it.

As for the Tornado Emergency guidelines, I appreciate the need for the warning language to be internally consistent and also the need for it to be properly formatted for downstream communication. However, I also believe there is a danger that it become a "canned" product. After all, an emergency is an exceptional situation and it is probably more effective to communicate in an exceptional manner. I'm afraid if it becomes a canned product, it will be used way too often and lose its urgency, as "regular" tornado warnings have.

I don't think quibbling over the language is the real issue. The real issue is to find a way to bring down the false alarm rate so warnings will be taken more seriously. That will take time, but I do hope there will be some emphasis on this in evaluation of WFO performance.

 

[Patrick Marsh]

I don't think quibbling over the language is the real issue. The real issue is to find a way to bring down the false alarm rate so warnings will be taken more seriously. That will take time, but I do hope there will be some emphasis on this in evaluation of WFO performance.

Although I agree with your statement on the FAR issue, let me play devil's advocate here. The Joplin Service Assessment, and numerous social scientists at the Weather Ready Nation workshop, stated that there was no FAR issue.

 

[rdale]

Be careful though as I think blanket FAR claims miss the point. The NWS may have nailed the warning, but county-based sirens and text messaging spread the wealth farther than they should.

If my county gets 1 tornado warning a year, am I really going to say "Well two years ago they missed, so I'm not going to shelter this time?" No. But if I get 29 in a day (Madison Co AL) then maybe I do ask that question...

And TORCON is a nice concept, just not so sure it does much for the public. It does a lot for TWC though

http://www.nynjpaweather.com/2010/06/07/my-thoughts-of-dr-greg-forbes-torcon-index/

 

[Jeff Duda]

I don't think quibbling over the language is the real issue. The real issue is to find a way to bring down the false alarm rate so warnings will be taken more seriously. That will take time, but I do hope there will be some emphasis on this in evaluation of WFO performance.

I disagree. I don't think you're going to see any significant reduction in FAR from the NWS over the next several years. I've said this before and I'll say it again: even though I'm not with the NWS so I don't know this for sure, I'm pretty sure the NWS aims for high POD almost regardless of the FAR that comes with it. This is because research has shown that people consider being killed by severe weather without warning to be one of the worst things possible.

That said, I think improving how the gravity of the situation is conveyed by improving the text part of a warning (assuming no other major changes to the warning system occur) is at least as important as reducing FAR in warnings, and likely more important.

 

[Mike Smith]

The Joplin Service Assessment, and numerous social scientists at the Weather Ready Nation workshop, stated that there was no FAR issue.

Hi Patrick,

I don't know which of the social scientists you were speaking with at the WRN workshop, but Dr. Laura Myers believes false alarms are a major issue. I discussed it explicitly with her. Also, my reading of the JLN SA does not lead me to the conclusion there is "no" FAR issue. To the contrary, page 7, for example, states:

It was common in the interviews to hear residents refer to ―storms always blowing over and missing Joplin, or that there seemed like there was a ―protective bubble around Joplin, or ―there is rotation all the time, but never in Joplin.

That certainly tells me there is a false alarm problem.

Mike

 

[Patrick Marsh]

Hi Patrick,

I don't know which of the social scientists you were speaking with at the WRN workshop, but Dr. Laura Myers believes false alarms are a major issue. I discussed it explicitly with her. Also, my reading of the JLN SA does not lead me to the conclusion there is "no" FAR issue. To the contrary, page 7, for example, states:

It was common in the interviews to hear residents refer to ―storms always blowing over and missing Joplin, or that there seemed like there was a ―protective bubble around Joplin, or ―there is rotation all the time, but never in Joplin.

That certainly tells me there is a false alarm problem.

Mike

First, let me state that personally I believe their is a problem. Hence the figures I created for Harold for his presentation. Hence the blog posts I've been working on for my website. Hence my upcoming talk on Tornado Emergencies at the National Severe Weather Workshop in March. I'm trying to bring attention to this issue.

However, in playing devil's advocate, I'm echoing the comments of several social scientists who used Eve Gruntfest's work in defending that there is no FAR problem. I don't know their names (except Eve's). I also seem to think I heard it in one of the presentations on Tuesday.

As for the Joplin Service Assessment, if you go past your quote, page 8 has this nugget:

It should be noted that stakeholders in the warning process, such as media or Emergency Managers, were less likely to think over-warning or desensitization to NWS warnings was an issue. During interviews, Emergency Managers in particular felt the frequency of warnings was appropriate, while media staff were split with some saying ―most warnings were `cry wolf', while others emphasized the importance of advance warning for all tornadoes regardless of false alarms

And throughout the Assessment the document doesn't actually say there is a FAR problem. It always prefaces that with the word "perceived". If the NWS actually thought there was a problem, wouldn't they have dropped the word "perceived".

Most importantly, the perceived frequency of siren activation in Joplin led the majority of survey participants to become desensitized or complacent to this method of warning.

perceived frequency of siren activation in Joplin,

the perceived frequency of siren activation (false alarms)

for residents in Joplin who, based on their perceived frequent exposure to local warning systems (and NWS warnings) during spring

the perceived frequency of siren activation

The text you quoted, Mike, actually tells me nothing regarding a false alarm problem. At no point in that quote is a warning actually mentioned. What it does tell me is that Joplin residents have a fundamental misunderstanding of our atmosphere. They don't think it could happen to them. They suffer from normalcy bias.

If you have a chance to talk with a certain regional director from the south, be sure to ask him what he thinks about a "FAR problem"... He'll be sure to set you straight...

 

[Simon Andersen]

GREAT points Patrick, I was trying to formulate a view like yours but didn't know how to, and you hit the head on the nail. I wish the thanks-feature was still around.

 

[Mike Smith]

Patrick,

Actually, as you correctly point out, half of the members of the media do believe there is a false alarm problem.

I know what normalcy bias is but I do not believe the SA handled that topic well. There was (still is?) a false alarm problem in Joplin due to the combination of, since '73, sounding sirens for SVR if winds ≥70mph + SGF's too-frequent tornado warnings for Jasper Co.

People do not want to believe they or their "tribe" (to use one social scientist's characterization) is incorrect. So, they "rationalized" the frequent siren activations (all false alarms until the second activation May 22) rather than questioning whether the siren activation policy was optimal. We were inadvertently training them not to act when the sirens sounded.

As I discussed last week, the "failure to act" issue is related to the majority of the public having little or no faith in our warnings. Saturday, I cross-posted my blog's post about the upcoming blizzard on a railroading blog because lots of members live in the affected area. The post immediately after mine said:


While it is nice to wish for such storms, I doubt that this is anything more than wishful thinking by the author.

I have seen nothing on any NWS sites that indicate that anything of this sort is forecast for the areas mentions. And, I live there!

Of course, a fierce blizzard occurred and -- we are learning now -- hundreds of cars are stranded.

The NWS did a fine job on this storm yet some people did not pay attention. I continue to believe part of the problem is our failure to tell people of the greatly improved quality of our warnings.

I plan a posting on this topic on my blog tomorrow.

Thanks for the good discussion, Patrick.

Mike

 

[Patrick Marsh]

So, if people think there is a false alarm problem, the media is split 50/50 (ish), and the emergency manages don't think there is a problem, then is there a problem? The NWS continues to get a mixed-message!

As I said, I think there is a FAR problem and I'm deeply disappointed that the Joplin service assessment didn't go far enough in addressing this. I know I brought up several times wanting to stress the FAR issue more in this assessment only to have other issues take precedence. It's sad! This is an issue that needs to be addressed!

Regarding the blizzard, although some parts of the storm were well forecast, other parts were not. How about the Blizzard Warning from Amarillo, for Amarillo (the largest city in their CWA by a longshot!). It went into effect at 6AM, while the 12 UTC sounding clearly showed that a lot was going to need to change for snow. (It was 49/49 at the surface, and very warm aloft!) It was still raining in Amarillo at 4PM with a surface temperature around 40F! That's at least 10 hours where a blizzard warning was in effect and nothing more than rain was falling! What if you shut down schools based on this forecast? It would have been a complete waste! Forecasts the morning of the event called for 6-9 inches of snow in Amarillo. They got 1".

Yes, forecasts are much better than they were. Yes, people in Kansas who didn't heed the warnings paid a terrible price, however, those in Texas who didn't heed the warnings and got away with it. Next time WFO Amarillo issues a blizzard warning for Amarillo, what will residents do...

Yes, we should tout our improved forecasts, but we shouldn't discredit our failures. We should be sensitive of them and demonstrate that we learn something from them. Tying this back to the FAR problem, the NWS does a great disservice to the field by ignoring the FAR issues. Yes, we are better than we were 10 years ago, but we have a long ways to go.

As always, thank you, too, Mike, for continuing the discussion.