Stoking the Fire Between Public and Private Sector Weather

[Steve Miller]

Extreme weather is increasingly seen as an economic liability, and a desire to get additional weather data beyond that provided by NOAA has turned into a $7 billion private weather forecasting industry. NOAA remains the sole authorized issuer of severe weather watches and warnings, but the new batch of private-sector competition means that the federal government soon may no longer be the sole source of the raw data that has fueled forecasts from local TV to agricultural to aviation.

Andrew Freedman, The Washington Post

 

[rdale]

"NOAA remains the sole authorized issuer of severe weather watches and warnings "

To clarify the article - NOAA is the only government authorized issuer. Anyone can issue their own as long as they don't represent them as official government watches and warnings.

 

[Jeff Duda]

No doubt that some (larger) private companies have long been supplementing the taxpayer-funded major data sources for many years already, but it will be interesting to see how things continue to evolve as more private companies sprout up and begin providing yet additional (quality) data sources. I suspect at least for the short term (up to the next 10 years) NOAA/fed will continue to pay these private sources for access to their data, but I also believe NOAA will continue to be the single-largest aggregator of meteorological data (at least for the US) for quite some time.

The fundamental fact remains that many private companies built off the work and products performed by taxpayer-funded entities (scientists and resources). So if anyone pushed too hard to hold NOAA out of some kind of large-scale meteorological service, they would probably find out pretty quickly how hard it would be to provide services for their clients without data and models provided by NOAA. WRF will only continue to be supported for so much longer before the money likely runs out. After that, I'm not sure which "free" NWP model will be supported by the research community. I'm not yet convinced it will be FV3.

 

[Jeff House]

I felt like the public private divide had narrowed/improved several years ago. I'm not sure why the fire is stoked, or the doberman is poked.

Most business-to-business B2B weather vendors agree that NOAA NWS is the best source of public warnings. Said alerts might be delivered via private sector, but the forecast and warning is 100% NWS.

The same B2B vendors rightly assert that businesses have specific criteria that might differ from those of the public. For instance trains don't care much about large hail. Some manufacturers are as sensitive to lightning as other severe wx. The best use of scarce resources (Econ 101) is for the private sector to handle those B2B issues. Yes they use NOAA data. Just like so much of business uses the taxpayer funded highway transportation system.

I worked for two B2B weather vendors in the past. I also believe the public needs a single source - NWS. Hopefully I did not tick off both sides. I just liked that harmony from several years back.

 

[Steve Miller]

Jeff, I too was under that impression (from the B2B side) and I'm disappointed the author has a different opinion and a wide audience with whom to voice it. Hope things are not so dire.

 

[rdale]

I don't think it's "dire" - as we know, headline writers are not usually the author But clearly the private sector is advancing FAR faster than the public sector. The NWS has ways to automatically update polygons by the minute (FACETS Threat In Motion) but at a minimum that's two years away. If the private sector comes up with that service and an automated warning app, I could see many people "jumping ship" to those warnings and away from the county-based NWS process.

IBM's Deep Weather doesn't care about county warning areas, siren activation policies, or forecaster philosophies. Would you pay $5 a month in the year 2025 to get those alerts? Maybe...

 

[Jeff Duda]

IBM's Deep Weather doesn't care about county warning areas, siren activation policies, or forecaster philosophies. Would you pay $5 a month in the year 2025 to get those alerts? Maybe...

The big assumption here being that Deep Weather provides statistically more accurate forecasts, which is far from a given. Right now this seems more like a flashy-trashy gimmick than any sort of step increase in forecasting accuracy.

 

[rdale]

Right now that specific example is very likely a gimmick. But does anyone believe the NBM is the best that can be done? Adding new models to the mix - and even FACETS - is not revolutionary in the way I think Machine Learning is going to offer...

 

[Jeff Duda]

As a non-federal employee and developer of NWP models, I do not endorse NBM as the end-all-be-all of forecasting. I see that as just a mish-mash of what we have available, and scale issues aside, many individual models can outperform NBM at any given time, although not consistently.

RRFS in the early 2020s will hopefully be the first truly unified forecast system used operationally, and will hopefully solve a lot of problems shown by the various models over the past 10 years.

 

[Alex Elmore]

Of course, we can't have an article comparing NOAA/NWS to any other forecast service without bringing up "the Euro nailed Sandy".

Having worked closely with NWS offices over the last few years, I'd say this article paints a bleaker picture of the situation that what is really going on. Everyone knows their place and they stick to it. Private companies can provide whatever type of forecast they want to airports, railroads, etc., and the NWS sticks to the general public, Emergency Managers, and other government bodies.

One specific part of the article I found controversial was the comment about the government basically seeming uninterested in improving the NWS, never mind a different Washington Post article covered the government passing the first legislation in decades to improve public forecast services back in 2017: https://www.washingtonpost.com/news...ensive-weather-forecasting-and-research-bill/

If the private sector comes up with that service and an automated warning app, I could see many people "jumping ship" to those warnings and away from the county-based NWS process.

Right now that specific example is very likely a gimmick. But does anyone believe the NBM is the best that can be done? Adding new models to the mix - and even FACETS - is not revolutionary in the way I think Machine Learning is going to offer...

I'm not sure if the FACETS Threat in Motion/Probabilistic Hazard Information can be considered full automation, as forecasters have some control over that process. The NWS's main "customers" are EMs and the general public, and while the general public will obviously have their attraction to services that do provide apps and push notifications, most EMs have a strong connection with their local NWS offices. When it comes to high-impact weather, there is a lot of communication that goes on between NWS forecasters and EMs behind the scenes that goes much deeper than a polygon or watch area.

Sometimes the least "sexy" options work the best in certain situations. While the NBM is not necessarily the product of a machine learning algorithm, it is produced in a similar way in its most basic sense: take a large amount of data, and condense it down into a presentable, easily "consumable" final product. It allows forecasters to not spend as much time during that time frame and focus more on the short-term forecast.

Machine learning and automation are tricky things and need to be used carefully in the weather enterprise. The training data, algorithms, thresholds, etc. that are used are still determined by and touched by humans; thus, there is still room for error. There still needs to be a human forecaster that sits between the output and the customer/public/whomever to quality control and interpret.

 

[rdale]

Having worked closely with NWS offices over the last few years, I'd say this article paints a bleaker picture of the situation that what is really going on. Everyone knows their place and they stick to it. Private companies can provide whatever type of forecast they want to airports, railroads, etc., and the NWS sticks to the general public, Emergency Managers, and other government bodies.

The gist of the article though is that "their place" may be shifting The NWS refuses to provide an app, so the public very rarely gets that forecast anymore. They'll get TWC or AW if they use their phone. In 2025 if those providers are offering Tornado Warnings that are higher quality than the NWS, why wouldn't the public want to use that source? That's where the confusion could come in because I doubt TWC and AW would coordinate on a warning issuance

One specific part of the article I found controversial was the comment about the government basically seeming uninterested in improving the NWS

But it's incremental. There's nothing revolutionary in the next 5-10 years for the way NWS puts out forecasts. Private sector providers are not stuck with a 20 year implementation program timeline (AWIPS2 )

I'm not sure if the FACETS Threat in Motion/Probabilistic Hazard Information can be considered full automation

It's not. But once the private sector is ready to move ahead with fully automated warnings, they can do it.

When it comes to high-impact weather, there is a lot of communication that goes on between NWS forecasters and EMs behind the scenes that goes much deeper than a polygon or watch area.

So when the NWS is issuing a SVR for a storm, TWC is issuing a TOR, and AW says it's just heavy rain - that could be confusing. Waiting until it happens - which seems to be the current plan - seems to be setting things up for a rough ride...

There still needs to be a human forecaster that sits between the output and the customer/public/whomever to quality control and interpret.

Maybe I'm overly optimistic about the future, but I have a hard time believing that in 2029 there has to be a human in the mix for the daily forecast... let alone warnings...

 

[Alex Elmore]

The NWS refuses to provide an app, so the public very rarely gets that forecast anymore. They'll get TWC or AW if they use their phone. In 2025 if those providers are offering Tornado Warnings that are higher quality than the NWS, why wouldn't the public want to use that source? That's where the confusion could come in because I doubt TWC and AW would coordinate on a warning issuance

I wouldn't say they refuse to do so, and it goes well beyond $$$ and effort. Again, contrary to this article and what some believe, the NWS and private sector do mostly stay in their lane. That's why you won't see the NWS providing direct support to stadiums, railroads, airports, etc., nor do you currently see the private sector issuing warnings to the general public. If the NWS produced an app, it would have to be free (people already pay taxes for it), which would undermine the private sector, and would ideally use location-based services, which for the purposes of weather forecasting, are copyrighted by a private sector company (I think). Regarding the warning comment, that is exactly why it should be left to one source, and I believe those in power see it that way too. Private companies can and do issue custom warnings to paid users, which are going to be other private businesses, airports, etc. That's perfectly fine, because while they're doing that, the NWS is providing information for the general public and various gov. officials. Which leads to my next point...

So when the NWS is issuing a SVR for a storm, TWC is issuing a TOR, and AW says it's just heavy rain - that could be confusing. Waiting until it happens - which seems to be the current plan - seems to be setting things up for a rough ride...

Yes, it would be confusing. However, the NWS has been building relationships with what they call core partners (EMs, public safety officials, etc.) through regular communication, both in person and remotely. This is done to understand what information these partners need in certain situations, which is the backbone to the NWS's Impact-Based Decision Support Services (IDSS). This is a huge program within the NWS, and has received great positive feedback from those core partners who use it. Based on what I've seen and heard, I don't see them "jumping ship", nor do I see them looking at multiple sources at once, even if there were to even be another source of public accessible warnings, watches, etc. any time soon.

Maybe I'm overly optimistic about the future, but I have a hard time believing that in 2029 there has to be a human in the mix for the daily forecast... let alone warnings...

As someone who has designed and built a machine learning-based product for use by forecasters, I would not be comfortable getting all of my information, especially warnings, from raw machine output 100% of the time with absolutely no oversight. If that's where my 75 degrees and blue sky forecast comes from, fine. But if there's a tornadic QLCS heading for my neighborhood, I don't want an algorithm to tell me if I should seek shelter or not. That very well may just be me, and I'm fine with that. We are nowhere near being able to rely on complete automation to warn, whether severe or tornado, on a storm. The research community is still trying to determine the exact mechanism(s) behind tornadogenesis, and why some storms produce tornadoes while others don't. AI isn't going to figure that out overnight, because it's only going to be trained upon the data we give it. If we don't have the proper data, then it's not going to have a perfect or near perfect hit/miss rate. Decades in the future, maybe. 2029? Unlikely. And if I'm wrong, then I will gladly come back to this post to admit it .

 

[rdale]

That's why you won't see the NWS providing direct support to stadiums,

Speaking as someone in the private sector who had a weather support agreement with a large college football stadium that ended when the local WFO came in and said they would provide two mets on-site for each game at no cost, that line can be fuzzy

If the NWS produced an app, it would have to be free (people already pay taxes for it), which would undermine the private sector

I don't think that would undermine anything, as long as the product provided wasn't "value added." AW and TWC are free and people still download those when they already have Siri and Apple's default app. Shoot, even something as simple as an agreement with Apple to provide the data source for their default weather forecast would be good. But for some reason Apple isn't interested in NDFD output

Based on what I've seen and heard, I don't see them "jumping ship", nor do I see them looking at multiple sources at once, even if there were to even be another source of public accessible warnings, watches, etc. any time soon.

I would agree the core partners won't (most of them - I still shudder when I see EM's share the local high school kid's forecast but that's another story) but we've seen what happens when the private-sector warnings "leak" into the general public (Virginia university, Kansas turnpike, etc.) I think the article points out that we can't keep blinders on and say "private sector warnings are getting to the public more and more, but over time it won't get worse."

And if I'm wrong, then I will gladly come back to this post to admit it .

At least we agree ST will still be here

 

[Steve Holmes]

The NWS refuses to provide an app, so the public very rarely gets that forecast anymore.

The WeatherBrains podcast folks say pressure from private-forecast providers, particularly AW, is keeping the NWS from having an app. WB folks go ballistic over this.

Maybe I'm overly optimistic about the future, but I have a hard time believing that in 2029 there has to be a human in the mix for the daily forecast... let alone warnings...

There will always be a place for a human in severe-weather coverage. When things get nasty, we humans want to get the lowdown from another human. People who follow specific TV mets long enough can tell the difference in tone between "we have to be on the air for this" and "this is the real deal." One met reported people picked up on the seriousness of the situation when they heard furious typing in the background. It's well known in Birmingham that, when James Spann removes his suitcoat, the S is about to hit the F. We can't get that from a computer.

clearly the private sector is advancing FAR faster than the public sector.

I'm trying to remember the last significant invention to come from government. Even with the space program, which is mostly government, it employs private contractors. The private sector can get a product to market far more quickly than can a government agency. If a business comes up with a vastly improved tornado warning (which I expect), the licensing issues could get tricky. It will be created with NWS data.

I wish them well. I understand the need to follow the NWS lead; extra minutes spent forecast-shopping can claim lives. But as a "Question Authority" guy, it's tough to watch everyone fall in lock-step behind an agency whose warnings are wrong 75% of the time.

 

[rdale]

The WeatherBrains podcast folks say pressure from private-forecast providers, particularly AW, is keeping the NWS from having an app. WB folks go ballistic over this.

You're probably right. But... Since when does the government have to submit to pressure from a private sector company? NASA has an app and I don't see SpaceX complaining

There will always be a place for a human in severe-weather coverage. When things get nasty, we humans want to get the lowdown from another human.

Oh absolutely. I can see big markets still having a few mets at TV stations for that, while most will use the Sinclair model where someone in a national HQ office does the livestream. But for most people, they'll be at a point where the app does most of the work, especially in less-than-tornado situations. Nobody pays attention to SVRs, but if they did get an alert saying 2" hail is coming to their home in 13 minutes and it will last for 3 minutes - I think that's going to be helpful.

If a business comes up with a vastly improved tornado warning (which I expect), the licensing issues could get tricky. It will be created with NWS data.

There wouldn't be any special "tricks" at play. Once the NWS provides the data, you can do whatever you want with it. It's no different already with forecasts. I use almost exclusively NWS sources when I create my forecast, but I don't have to "license" anything.