New Forecast Soundings Online - SHARPpy

[Greg Blumberg]

For those unfamiliar with the recent updates in online weather data and have been living under a rock for the past few months, this is for you:

HOOT, the student-run organization that develops weather products at the University of Oklahoma, has begun releasing new forecast soundings for public consumption. As of right now, HOOT is only offering select forecast soundings from the RAP, although plans are in place to expand the available forecast soundings to a wide variety of locations across the U.S., and for many different numerical models.

Those familiar with other online weather data sources may recognize that these soundings look very similar to SPC's observed soundings online. These soundings are plotted using a sounding plotting and analysis package Kelton Halbert and I have been developing called SHARPpy, which is a Python-based derivative of SPC's NSHARP program we inherited from Patrick Marsh. Since we started managing the code last March, we've made significant changes and have implemented a large set of analysis routines previously internal to SPC. We hope that having these unique routines available online will enhance your forecasting! Check them out!

http://hoot.metr.ou.edu/models/rap/fsound/
http://hoot.metr.ou.edu/upperair/

Here's an example sounding from a while back:

 

[Jeff Duda]

Would really like to see some moist adiabats on the skew-T. The indicated parcel may not always be the one that becomes a thunderstorm, and right now there's no way to assess the stability of other parcels with any certainty.

 

[Rob H]

So are these changes going to be made to the sharppy repos I see on Github?

 

[Kelton Halbert]

Would really like to see some moist adiabats on the skew-T. The indicated parcel may not always be the one that becomes a thunderstorm, and right now there's no way to assess the stability of other parcels with any certainty.

Jeff,

There is code already in there that draws the moist adiabats. It's just not turned on. We turned it off because we were starting by getting as close to the SPC Observed soundings as possible, which have no moist adiabats, only the most unstable parcel trace. And there's actually plenty of ways to assess the stability of other parcels, including the thermodynamic window that has information on the most unstable, surface based, mixed layer, and forecast surface parcels.

As far as your reason for wanting it, it should be mentioned that there is a cross-platform GUI/Desktop version that can be run from any Mac/Windows/Linux home computer. This portion is still under development, but it would allow for you to choose (or even define your own) parcels and display them. You could also choose to show or hide the moist adiabats.



Rob,

The current version is sitting in a private repository on github right now. We will be making the code open source at AMS 2015 in Phoenix, and it will be freely available for people to use and download. We're just keeping the github closed for now while we work out the remaining kinks.

 

[Kelton Halbert]

I should also mention that this has ensemble capabilities as well. This is a much older image from when things were not as near complete as they are now, but it is entirely possible to display SREF/other ensembles in the same framework.

 

[Rob H]

That is a really slick looking app (presentation and code), which is something I don't say very often with weather software. I'll have to re-asses if I want to continue working on my pet project Skew-T now

Kudos to you guys for using an appropriate language, great source control, and making a version available to the public (eventually)!

 

[Greg Blumberg]

Thank you for the compliments Rob! It's been a labor of love for both of us, and we have been overwhelmed by the support we've gotten from the meteorological community on this project. Any positive response we get makes use more thrilled to be contributing this to others. :

Jeff - thanks for the input. We'll now be adding a moist-adiabats show option to the GUI program.

 

[Kelton Halbert]

I took some location suggestions over Twitter and have updated the soundings page to include them. Obviously their suggestions were geographically biased, so feel free to suggest some other locations.

The current method of selecting a sounding site is not the permanent solution. We are working on a much easier-to-use point-click system, but in the meantime, the menu selection method is the one in operation. Because of this reason, we cannot include every point-click site that we have available, but we can try to cover the big ones.

 

[Jeff Duda]

Jeff,

There is code already in there that draws the moist adiabats. It's just not turned on. We turned it off because we were starting by getting as close to the SPC Observed soundings as possible, which have no moist adiabats, only the most unstable parcel trace. And there's actually plenty of ways to assess the stability of other parcels, including the thermodynamic window that has information on the most unstable, surface based, mixed layer, and forecast surface parcels.

Perhaps I did not adequately describe the suggestion I had. I'm aware that you can look at scalars such as CAPE and CIN of the SB, ML, MU and "FCST" parcel (although I'm not sure how the "forecast" parcel is determined), but there are a lot of ways to integrate Tenv-Tparcel between the LFC and EL to get 2300 J/kg of CAPE, for example. All I'm saying is that since you guys made the effort to display detailed profiles, I see no reason not to keep going and make it as exquisite as possible. Adding just a few moist adiabats (say every 10 C) would be one way of doing that. Another way is by adding full parcel paths for the SB, ML, and MU parcels, although I could see that cluttering the image. But if this is going to be available on a website, perhaps a check box or radio box could be used to select a specific parcel, with that specific parcel path then plotted alone.

As far as your reason for wanting it, it should be mentioned that there is a cross-platform GUI/Desktop version that can be run from any Mac/Windows/Linux home computer. This portion is still under development, but it would allow for you to choose (or even define your own) parcels and display them. You could also choose to show or hide the moist adiabats.

I eagerly await the release.

 

[Rob H]

Can anyone explain what a storm slinky is? Never seen that before, and my googling doesn't turn up much useful.

 

[MClarkson]

much easier-to-use point-click system

That is basically the holy grail of sounding programs. Especially if it keys off the nearest model grid(or interpolates between the nearest groups) instead of grabbing the nearest defined sounding(say an airport).

 

[Kelton Halbert]

Can anyone explain what a storm slinky is? Never seen that before, and my googling doesn't turn up much useful.

Rob,

We had a very similar problem when trying to write it up ourselves as well! It's something that is used in the SPC internal version of NSHARP, and is essentially a 2D parcel trajectory. We're working on having a documentation website to explain such details, but to answer your question on the Storm Slinky...

This is directly from the Department of Commerce/NOAA AWIPS 2 training page, and this information is consistent with direct conversations with people over at SPC.

The storm slinky represents a 2D view from above of the lifted parcel trajectory after a 5 m/s nudge at the Level of Free Convection (LFC). After the parcel is nudged, the derived buoyancy accelerates the parcel to the Equilibrium Level (EL). The spacing of the rings is proportional to the horizontal displacement during a fixed time, and the color coding matches the hodograph. Kidney bean shapes are common with classic supercell wind profiles, while other unusual slinky shapes may suggest unfavorable aspects of the wind profile. Some aspects of the display include:

white line - storm motion direction

circle color - parcel circles are color coded relative to height coincident with the colors on the hodograph

deg - updraft tilt off of vertical calculated using the angle between the first ring and last ring (e.g. -104 deg). According to SPC, there appears to be an error in the calculations at this time.

 

[Greg Blumberg]

MClarkson. Thanks for your comment. Currently we have only selections from METAR stations, which given the density of the airports and the typical grid spacing of numerical models as of this point is sufficient for the time being. I often wonder how useful having access to soundings from all the grid points is given the fact that the grid points nearby can be highly correlated with the sounding location you select.

For the GUI application, we'd like to be able to have point and click off the model grid, but that requires downloading the whole model grids. We experimented with that at first, but found that downloading the grids is too slow for use of the program. Also, they would take up too much disk space.

Here's an example from the observed sounding at Omaha, NE the day of the Twin Tornadoes in Pilger to illustrate what the AWIPS 2 documentation means by the kidney bean shaped slinky:

 

[MClarkson]

I often wonder how useful having access to soundings from all the grid points is given the fact that the grid points nearby can be highly correlated with the sounding location you select.

I agree for most of the US, it does not matter, the nearest airport is fine. That benefit mainly comes in outside the US or in some of our sparsely populated western mountain regions.

but that requires downloading the whole model grids. We experimented with that at first, but found that downloading the grids is too slow for use of the program. Also, they would take up too much disk space.

I feel like the grib/ungrib utilities are pretty slow compared with what could be done in one of the heavy hitting data processing languages. I am about to start working on a large-domain high-speed grib decoder in C, I'll let you know if I make any process. You wouldnt be able to archive much on a standard machine, whole domains taking up too much space, but it might work from a pure forecasting standpoint where you only need a couple runs.

 

[Rob H]

I feel like the grib/ungrib utilities are pretty slow compared with what could be done in one of the heavy hitting data processing languages.

From the source code I've looked at, performance is left on the table in a lot of weather software because many (not all) meteorologists aren't developers first and foremost and don't put enough time into it to care about performance tuning when something works already. They learned FORTRAN and BASIC back in 1982, found some patterns that worked, and then shoehorned everything else into those patterns. I won't hold that against them because they're not supposed to be amazing developers. It'd be like if someone forced me to start forecasting to the public - I'd do it the way I learned, inefficiently and resistant to change.

I honestly don't think you need to go as low level as C, unless that's what you love working in (and I'd call you a masochist) - we have stuff at work that processes 70,000 updates/second for financial data written in C#. I like this trend of meteorologists writing in modern languages like Python, and putting the code out on Github. I'm not going to rewrite degrib from scratch because I've got other stuff I'd rather work on, but if I see a way to speed up SHARPpy I can submit a pull request nice and simple.

 

[Kelton Halbert]

I'm not going to rewrite degrib from scratch because I've got other stuff I'd rather work on, but if I see a way to speed up SHARPpy I can submit a pull request nice and simple.

And this is exactly what we'd love to see happen. I know it's hard to get people on the code contribution bandwagon, but this is the hope and dream once the open source release happens.

Also, I'd love some input on what people would find intuitive for a user interface when selecting observed/model soundings from a GUI. I've attached an image from the current working version of the sounding selection system. Obviously this is intuitive to me (I mean, I wrote it), but that doesn't mean that this is intuitive to everyone else. I want this to be easy to use (unlike BUFKIT...). Ideas?

 

[rdale]

Finally took some time to install this so 1) very happy I did! and 2) sorry if I've revived a dead thread

1) Any more info on the Storm Slinky? Or a "full" documentation site?
2) Using Windows, sometimes when I use the scroll wheel to zoom it goes where I'm pointing, and sometimes (most) it just zooms towards right-middle side of the display. Just me?
3) Select map area shows a bunch of maps, but nothing happens when I select one.
4) If I select a model and select all forecast hours, the button at the bottom correctly changes to "Deselect All". If I change models, the "deselect all" button stays but nothing is selected so it doesn't do anything.
5) In the SREF, what is the "main" line depicting? I thought it was the mean, but it displays well outside of the members frequently.
6) Observed: Are non 00/12Z launches captured in the "latest"?

 

[Drew Terril]

Apparently I have been living under a rock. This is fantastic. Out of curiousity, how do we go about finding the desktop program for a linux based system. Not entirely sure exactly what to type in on my command prompt window after the usual commands to install a program (command prompt is the only way I install on my machine anymore).

 

[Tim Supinie]

Finally took some time to install this so 1) very happy I did! and 2) sorry if I've revived a dead thread

1) Any more info on the Storm Slinky? Or a "full" documentation site?
2) Using Windows, sometimes when I use the scroll wheel to zoom it goes where I'm pointing, and sometimes (most) it just zooms towards right-middle side of the display. Just me?
3) Select map area shows a bunch of maps, but nothing happens when I select one.
4) If I select a model and select all forecast hours, the button at the bottom correctly changes to "Deselect All". If I change models, the "deselect all" button stays but nothing is selected so it doesn't do anything.
5) In the SREF, what is the "main" line depicting? I thought it was the mean, but it displays well outside of the members frequently.
6) Observed: Are non 00/12Z launches captured in the "latest"?

#1: Last I heard, Greg was still working on the full documentation, but I haven't heard the status of that in quite some time.
#2: The zooming on the Skew-T currently needs some work. I've got an idea for how to make it better, but I'm currently working on re-designing the profile picker, and my effort will be directed there for the mean time.
#3: Yeah, that's not implemented (Kelton put a lot of things in there as placeholders for future ideas). It might show up in a modified form in the re-designed picker.
#4: Hmm, so it does. I might have to patch that.
#5: The program pulls from the Penn State Bufkit archive for the model soundings, and the solid line for the SREF is what is listed as the ensemble mean in that Bufkit file. I have seen it outside the members on one occasion, and that was an issue with the file pulled from the Bufkit archive.
#6: It looks like the "Latest" option does pull from the non-synoptic-hour launches.

 

[rdale]

Thanks for the details! (And I can't believe this is open source / free!!!)

I'll play around with #5 doing some comparisons with BUFKIT to see if that can help. Also might consider keeping an alternate source of BUFR data like http://www.meteor.iastate.edu/~ckarsten/bufkit/data/nam/nam_vih.buf in the mix in case PSU goes down temporarily?

Suggestion: Either "play/pause" keypress, or the ability to hold the cursor <> buttons and let it keep moving? Holding doesn't do anything different than a single press.

 

[Tim Supinie]

Okay, I've submitted a pull request with that patch in it.

A part of the re-designed profile picker will include much more flexibility with data sources, so I'll definitely keep the Iowa State link in mind as another potential source. Thanks!

Hmm, Greg, Kelton, and I are all on OS X, which will flip through them if you hold down an arrow key. I guess it's different between OSes, so it might be a good idea to force it to flip through them somehow.

I'm glad you're enjoying it, but I've only been heavily involved for the last month or so; I can't speak for Greg and Kelton, who've been working on it much longer. I imagine they'll be glad too, though!

 

[rdale]

Thanks - I just got the note from Git. One suggestion - if I don't click "Latest" on the obs it will give me an error code. Can it just default to latest without needing to click?

More on the <> -- if I hold it does two slides. When I let go, it's at some far distant graphic. For example, on GFS, my first button hold shows me F03 and F06. When I let up, it's at F081. I repeat going backwards and it goes to F78 and F75, and then on release I'm at F141.

For the "would be nice" request list, it'd be nice if I could set a default for the obs, and another for the models. I'm primarily always starting with the same (DTX/LAN) so it'd avoid map scrolling and picking.

And not that it was hard to do - but I'll probably put together a video walking through the setup for Windows users this week.

 

[rdale]

Any pointers for intro to Python that would apply here? Any chance of requesting "day of week" shows up in the timestamp up top for forecast models?

And I don't see that bug #4 was fixed.

 

[Andy Berrington]

And not that it was hard to do - but I'll probably put together a video walking through the setup for Windows users this week.

This is a good idea, in addition to one for OS X users.

I hit a few snags in the setup personally, so I haven't been able to get it running yet.

 

[Greg Blumberg]

Let me try to answer several posts at one time. Tim was right, I am thrilled that everyone is enjoying this.

Rob Dale,

Much of what you suggested regarding the "latest" and default observation will probably be fixed with the redesigned Sounding Picker Tim's working on. I'm not certain when we can expect that to be done, as Tim said there's a lot of flexibility we're trying to incorporate into that portion of the program. We want to make it easier for others to add sounding sources to the program, as we have multiple parties interested in using the program such as folks at NOAA/NCAR/universities/etc. We really appreciate your extra help in figuring out the BUFKIT files and the installation tutorial. We're just three OU students putting this thing together right now, so things like this help us push the program past the beta phase.

The full documentation will probably be the next thing I start working on. For now you can use the SPC documentation for their sounding page and the SHARPpy README together. Combined both of those should cover at least 90% of the available features in the SHARPpy GUI as much of the program mimics the functionality you'll see on the SPC observed online sounding images:

http://www.spc.noaa.gov/exper/soundings/help/index.html

I think right now our primary goal is to get the program past the beta status, which many of our beta testers have really helped out here.

Drew and Andy...I just updated the README to be a little more clear about the steps to install. You will need to be familiar with command line to install this, but the number of steps is quite simple. We have plans to try to make this a portable executable so you don't have to go through all the steps with the command line to install the program, but that's another item on the never-ending list of things to do with the program.

For all, here's a link to the program:

https://github.com/sharppy/SHARPpy
http://sharppy.github.io/SHARPpy/index.html