Which models suck and why?

[Stan Rose]

WOW! I would like to give a huge thanks to Jeff for his detailed and thorough breakdown of the complications of dealing with models. I've been reading a lot about these factors during my studying, but Good Lord! I read his post and thought that it's amazing that anyone besides the most brilliant scientists could look at models and get anything reliable out of them! It makes me wonder how chasers are really using model data. Don't get me wrong... many chasers have years and years of experience and often degrees (multiple degrees) in this field, and they know what to look for. But, the way it sounds, looking at the models and diagnosing the atmosphere would have to be your full time job to use them confidently. No wonder there are so many SPC chasers out there! I mean really... are most chasers cross examining a few models for basic parameters the day before and day of and driving out to a general area that looks good? Why not!? It seems like the best you could do without giving your life over to model study. Just check the NAM, GFS, HRRR, and SREF for upper level pressure and vorticity, 500 mb winds, 850 mb moisture and winds, soundings, CAPE, LI's, TT, SRH, Hodos, surface obs, and satellite... Then take your dart and throw it at the map on your wall and you're ready to go!

Well, I think it's becoming increasingly obvious that chasers are relying more and more on the high-res models, and this will be the wave of the future. The HRRR is the classic example. Two years ago, or even last year for that matter, I had little faith in it and while I looked at it I rarely relied on it in isolation for decision making. Recently, I haven't a clue what the developers are doing to tweak it, but it's obviously wayyyy improved this year. For the first time I relied heavily on early morning runs and in several cases, the model was spot on. And evidently other people recognized the trends too cause I saw more chasers arrive early at very specific spots that they wouldn't have chosen otherwise. Of course, it hasn't been exactly right all the time, but it's headed in the direction where these sort of models are going to be serious game changers. The ND tornado (Watford 5-26) is a classic example. Unfortunately, the HRRR only goes out 15 hours on a delay, so no time to race from TX (where all the chasers were that day, including yours truly) to ND in 12 hours! SPC didn't even have a slight risk up there. Not even a 2% tornado--nadda! But if you had used the 12Z (or 13Z...or...) HRRR run that day it would have been a no-brainer to target that area, and you would have placed yourself no more than 5-10 miles from where the tornado hit. Imagine that run being available 2 days in advance! It will be a game changer, im telling you--a much bigger influence on the future of chasing than cell phones or whatever. Come back here 10 years from now and ill say 'I told ya so!'

 

[MClarkson]

The downside of such a study is that the models are tweaked at least once every two years, if not more often.

All the more reason to automate the process. A one-off manual study might be outdated, but if you have the code that just sits there computing once per day, for example, the previous 3 months error stats... its a simple matter of waiting for the sample size to return after a major update. You don't even have to do any more work that way, unless NCEP changes the names of the file, in which case you'd have to do like 3 minutes of work.

 

[Dan Robinson]

Stan, I feel the same way. I think the short term high-res models (specifically the Updraft Helicity plot) will be all that is needed to pick a chase target in the next 5 years or so. I'm wondering if there will be any advantage to being experienced or having advanced (or even basic) forecasting knowledge in 5 to 10 years.

Right now, the HRR/4km WRF models still blow it enough that one can't yet rely on them. June 3 in Nebraska is one example. When they nail it though, they really nail it - and so they're definitely worth heeding even if one suspects they may be wrong on any given day. They'll only improve as time goes on.

 

[rdale]

A one-off manual study might be outdated, but if you have the code that just sits there computing once per day, for example, the previous 3 months error stats...

At that point seasonal differences come into play... It may not have handled moisture advection well in the spring, but nailed it in July. Yet your algorithm says it struggle... I'm not saying it wouldn't be valuable, just not sure how much value it'd have.

http://www.hpc.ncep.noaa.gov/mdlbias/

http://www.hpc.ncep.noaa.gov/html/model2.shtml#verification

 

[Paul Knightley]

What does suck is some people's ideas about what models should be expected to do!

 

[Rob H]

There are a lot of insightful details popping up in here, but since no one is directly answering the questions I started off with, and brought up again later, I'm assuming that means that you can't make blanket statements like the following:

The NAM overdoes moisture

even when you further define the question by saying "in the Central Plains, in late spring".

What I'm getting out of this thread is that you can't make statements like I listed out in the OP, and that every model needs to be independently evaluated within a certain time period, in a specific location, as a specific parameter. Any generic biases tend to be identified and corrected. And if you want to know how the NAM is handling moisture there isn't really a good way, other than digging into the data yourself. Is all that correct?

 

[MClarkson]

Any generic biases tend to be identified and corrected.

That one is not entirely true. Some of these are in regions of low priority to model developers, and other times some models are intentionally tuned to the performance of one specific field, at the possible expense of others. Also, some bias comes about, especially in courser resolution models, due to averaging over a grid square. That 1km wide mountain valley temperature will never be correct in the raw ~30km GFS or GEM output.

 

[Jeff Duda]

There are a lot of insightful details popping up in here, but since no one is directly answering the questions I started off with, and brought up again later, I'm assuming that means that you can't make blanket statements like the following:

The NAM overdoes moisture

even when you further define the question by saying "in the Central Plains, in late spring".

What I'm getting out of this thread is that you can't make statements like I listed out in the OP, and that every model needs to be independently evaluated within a certain time period, in a specific location, as a specific parameter. Any generic biases tend to be identified and corrected. And if you want to know how the NAM is handling moisture there isn't really a good way, other than digging into the data yourself. Is all that correct?

Go back to what I said in my lengthy post. You need to understand what's going on in a model to see how it might be mis-handling things. One thing the NAM seemed to do this spring was indeed overdo moisture. There was a fairly straightforward explanation for that, however. The NAM uses a monthly greenness fraction climatology in the land-surface model. Well, this spring the vegetation green-up was far from climatology. In fact, green-up was significantly delayed. So the model was assuming vegetation was more green than it actually was. Hence it was assuming too much ET and thus you would see too much near-surface moisture.

You just kinda have to know things like that. Unfortunately it's not always obvious or easy to determine things like this.

 

[Rob H]

Go back to what I said in my lengthy post. You need to understand what's going on in a model to see how it might be mis-handling things. One thing the NAM seemed to do this spring was indeed overdo moisture. There was a fairly straightforward explanation for that, however. The NAM uses a monthly greenness fraction climatology in the land-surface model. Well, this spring the vegetation green-up was far from climatology. In fact, green-up was significantly delayed. So the model was assuming vegetation was more green than it actually was. Hence it was assuming too much ET and thus you would see too much near-surface moisture.

You just kinda have to know things like that. Unfortunately it's not always obvious or easy to determine things like this.

I read your lengthy post and it was great, but only addressed the first part of what I was getting at.

To use your example, I'm making an assumption that every central plains forecaster and every SPC forecaster and every researcher needs to know that the NAM is overshooting moisture because it can affect day-to-day operations at their job. So is it assumed that these dozens or hundreds of people are all familiar with the NAM and recognized that the monthly greenness fraction climatology is off? There isn't a status product, or a bulletin, or something somewhere where the NAM gatekeeper said "hold up, this green up isn't happening the way it should" that disseminates that information to everyone that needs it? And no one goes in and tweaks it so that it's more accurate? It's up to each individual forecaster to notice this and account for it in their own way?

 

[rdale]

Well no there is no "status product." And you can't just go in and tweak climatology. Shortcuts have to be taken just because resources are not unlimited (well, maybe they are for the ECMWF production ) But yes, those of us who use forecast because it's our job and a daily responsibility know that. Just as we know that GFS at 240hrs in November will ALWAYS have a snowstorm, and in the summer it will ALWAYS have a Gulf hurricane.

Some of your education might be found at https://www.meted.ucar.edu/training_module.php?id=902

and the full set of NWP trainings at https://www.meted.ucar.edu/training...guageSorting=1&module_sorting=publishDateDesc

 

[rdale]

That being said - the closest thing you might find (other than AFDs in the area of interest) are the HPC model discussions.

http://www.hpc.ncep.noaa.gov/discussions/pmdhmd.html

 

[JamesCaruso]

Well, I think it's becoming increasingly obvious that chasers are relying more and more on the high-res models, and this will be the wave of the future. The HRRR is the classic example. Two years ago, or even last year for that matter, I had little faith in it and while I looked at it I rarely relied on it in isolation for decision making. Recently, I haven't a clue what the developers are doing to tweak it, but it's obviously wayyyy improved this year. For the first time I relied heavily on early morning runs and in several cases, the model was spot on. And evidently other people recognized the trends too cause I saw more chasers arrive early at very specific spots that they wouldn't have chosen otherwise. Of course, it hasn't been exactly right all the time, but it's headed in the direction where these sort of models are going to be serious game changers. The ND tornado (Watford 5-26) is a classic example. Unfortunately, the HRRR only goes out 15 hours on a delay, so no time to race from TX (where all the chasers were that day, including yours truly) to ND in 12 hours! SPC didn't even have a slight risk up there. Not even a 2% tornado--nadda! But if you had used the 12Z (or 13Z...or...) HRRR run that day it would have been a no-brainer to target that area, and you would have placed yourself no more than 5-10 miles from where the tornado hit. Imagine that run being available 2 days in advance! It will be a game changer, im telling you--a much bigger influence on the future of chasing than cell phones or whatever. Come back here 10 years from now and ill say 'I told ya so!'

I dread the day that this comes true. It would take a lot of the joy out of chasing for me. Part of the fun is figuring out the puzzle, and (to mix a metaphor) this would be like just going to the answer in the back of the book. A huge part of the challenge would be gone, as would the gratifying feeling of being in the right place at the right time because of your own skills. Gone would be the pride in having an understanding of the atmosphere that is beyond the grasp of the average person. No thrill of the hunt. Like gambling when you know the outcome of the game or the cards you're going to be dealt (without the monetary payoff ).

Maybe some of the veterans already feel this way relative to today's technologies that they at one time did not have, but it's a matter of degree; a big red blob that basically says "be here at 5:30PM CDT" is indeed a game-changing tipping point in a sad direction.

Jim

 

[JamesCaruso]

WOW! I would like to give a huge thanks to Jeff for his detailed and thorough breakdown of the complications of dealing with models. I've been reading a lot about these factors during my studying, but Good Lord! I read his post and thought that it's amazing that anyone besides the most brilliant scientists could look at models and get anything reliable out of them! It makes me wonder how chasers are really using model data. Don't get me wrong... many chasers have years and years of experience and often degrees (multiple degrees) in this field, and they know what to look for. But, the way it sounds, looking at the models and diagnosing the atmosphere would have to be your full time job to use them confidently. No wonder there are so many SPC chasers out there! I mean really... are most chasers cross examining a few models for basic parameters the day before and day of and driving out to a general area that looks good? Why not!? It seems like the best you could do without giving your life over to model study. Just check the NAM, GFS, HRRR, and SREF for upper level pressure and vorticity, 500 mb winds, 850 mb moisture and winds, soundings, CAPE, LI's, TT, SRH, Hodos, surface obs, and satellite... Then take your dart and throw it at the map on your wall and you're ready to go!

At least on the day of, the synoptic scale parameters depicted in the models are reliable enough for chasing purposes. Real-time surface obs then become more important for refining the target area.




Sent from my iPad using Tapatalk HD

 

[JamesCaruso]

Stan, I feel the same way. I think the short term high-res models (specifically the Updraft Helicity plot) will be all that is needed to pick a chase target in the next 5 years or so. I'm wondering if there will be any advantage to being experienced or having advanced (or even basic) forecasting knowledge in 5 to 10 years.

Right now, the HRR/4km WRF models still blow it enough that one can't yet rely on them. June 3 in Nebraska is one example. When they nail it though, they really nail it - and so they're definitely worth heeding even if one suspects they may be wrong on any given day. They'll only improve as time goes on.

Despite the improvements in modeling, isn't there an upper limit to the accuracy that can be achieved simply because we cannot (currently) get actual observations with enough density or frequency?

 

[JamesCaruso]

At that point seasonal differences come into play... It may not have handled moisture advection well in the spring, but nailed it in July. Yet your algorithm says it struggle... I'm not saying it wouldn't be valuable, just not sure how much value it'd have.

http://www.hpc.ncep.noaa.gov/mdlbias/

http://www.hpc.ncep.noaa.gov/html/model2.shtml#verification

Here is another interesting web page on model biases (Rob's link above took me to the interactive page, but it might have had a link to this one already):

http://www.hpc.ncep.noaa.gov/mdlbias/biastext.shtml