SN rolls out a Ranking system

[Tyler Allison]

http://www.spotternetwork.org/forum/index.php?showtopic=261

<asbestos shorts=on>

 

[Skip Talbot]

Are your points retroactively calculated from previous reports, or is everyone starting out fresh?

 

[Tyler Allison]

retro, at least from Jan 2010

you can see you own point calculation in the account menu area

 

[PKohlmeier]

SIGH... Seems I'll be stuck at where I am for awhile, I'm active when storms come through but it seems that so far our storms have been below severe strength here (luckily). I see that as one flaw for the ranking system at least from my perspective.

 

[Tyler Allison]

it's a drawback we are aware of. still trying to figure out how to address it.

 

[Jason Foster]

Tyler:

Seems like a decent idea. My first "gut" reaction would be this may promote unnecessary reports, like many folks reporting the same event, just to earn the points. Perhaps the system should be based on a ratio rather than actual reports. Also, it will be interesting who this will affect work load for you and others.

But again...this is just a first...gut reaction. I personally am not to concerned what my rating is really...the point is for the right people to get the information, but I see how this might help with habitual offenders.

 

[Ben Holcomb]

I haven't ever heard the National Weather Service complain they have gotten too many reports from a storm. I think that's a bridge that should be crossed if it becomes a problem

 

[Wes Carter]

I am curious, who determined if the reports are good, bad, or ugly? I see that Verne Carlson and David Drummond both have 'Very Bad Reports' in their list. However, even though I don't personally know either one of them I have seen enough of their work to be pretty sure they didn't report a "sheriffnado" or "extreme lightning".

Maybe it was a case of not providing enough information. That being said, when you are out chasing (especially if you are alone) sometimes all you have time to do is just click the boxes on what you see. There's no time to write and essay, no place to pull over safely to do so, it's hard to tell which way the tornado is moving or how fast, etc... But I've seen some reports given bad grades for not having enough detailed information. I thought that was the purpose of having the phone number on our profiles, so the NWS can call us for more information if they want it. It's a lot easier to talk into a Bluetooth headset while driving than to type out a detailed report. And some people might be less inclined to make reports if they are going to get dinged a point for not typing out that essay while trying to navigate safely through a tornadic supercell.

Edit: I just looked at my SN account. I got a red flag and a yellow flag for the May 10 tornado near Medford, OK. I wrote this while driving East on SR 11 with a possible rain wrapped tornado to my left. (So I really should not have typed anything.)
"approx 5 min ago tornado crossed sr 81 heading ene" I entered my position as 1 mile North of Medford, OK. I know that it would be great to have more detailed information, but see see the above. There were several reasons I didn't give more precise distance. 1. I was driving and the storm was moving at 60MPH. 2. It really is hard to judge distance. 3. The warnings don't get down to specific lots and houses, and the hook can be seen on radar. Is it going to have any benefit whatsover to say that it was 1 mile North of my position? If I overestimate or underestimate that and if hypothetically the NWS did alter their warning it could be more damaging than just issuing a warning based on reports and using the radar data to provide an estimated path. I thought the purpose of this grading system was to combat the people who are just throwing silly reports out, not to penalize people who are actually trying to do something good but sometimes don't add info because they don't have it or cannot safely provide it.

 

[Tyler Allison]

I am curious, who determined if the reports are good, bad, or ugly?

The QC team (names listed on our "About" page)

I see that Verne Carlson and David Drummond both have 'Very Bad Reports' in their list. However, even though I don't personally know either one of them I have seen enough of their work to be pretty sure they didn't report a "sheriffnado" or "extreme lightning".

Actually you bring up a good point. I used incorrect terminology and have since corrected it. "Very Bad" now reads "Low Quality".

Obviously, our quality control team is not there (usually) so reports are based on the "quality" of the report.

There are three categories:

- High Quality / Perfect Report [ green flag ]
- Acceptable Quality [ yellow flag ]
- Low Quality [ red flag ]

Maybe it was a case of not providing enough information. That being said, when you are out chasing (especially if you are alone) sometimes all you have time to do is just click the boxes on what you see. There's no time to write and essay, no place to pull over safely to do so, it's hard to tell which way the tornado is moving or how fast, etc... But I've seen some reports given bad grades for not having enough detailed information. I thought that was the purpose of having the phone number on our profiles, so the NWS can call us for more information if they want it. It's a lot easier to talk into a Bluetooth headset while driving than to type out a detailed report. And some people might be less inclined to make reports if they are going to get dinged a point for not typing out that essay while trying to navigate safely through a tornadic supercell.

Nobody is asking for an essay, in fact you might get a "low quality" report if you write an essay for no good reason.

If all you click is "estimated winds 50mph" you are going to get an Acceptable Quality report. It doesn't take but another 5 seconds to type: "1 inch tree branches snapped off". That makes it a High Quality report.

And let's turn this around. You can't expect the NWS to call every person who submits a report, especially to just clarify a small detail.

 

[Tyler Allison]

Edit: I just looked at my SN account. I got a red flag and a yellow flag for the May 10 tornado near Medford, OK. I wrote this while driving East on SR 11 with a possible rain wrapped tornado to my left. (So I really should not have typed anything.)
"approx 5 min ago tornado crossed sr 81 heading ene" I entered my position as 1 mile North of Medford, OK. I know that it would be great to have more detailed information, but see see the above. There were several reasons I didn't give more precise distance. 1. I was driving and the storm was moving at 60MPH. 2. It really is hard to judge distance. 3. The warnings don't get down to specific lots and houses, and the hook can be seen on radar. Is it going to have any benefit whatsover to say that it was 1 mile North of my position? If I overestimate or underestimate that and if hypothetically the NWS did alter their warning it could be more damaging than just issuing a warning based on reports and using the radar data to provide an estimated path. I thought the purpose of this grading system was to combat the people who are just throwing silly reports out, not to penalize people who are actually trying to do something good but sometimes don't add info because they don't have it or cannot safely provide it.

The SN advisors accept that we have not been consistent in the past. We have formed an internal group to come up with a consistent rating system.

It also takes 4 or more "red" flags to get ranked as a low quality report. That allows for variability of opinion between humans

-Tyler

 

[PKohlmeier]

Not to beat a dead horse, but since I feel SN is a great tool (if the local NWO used it) but like I posted on the support forum I think training tool on exactly how to report since it seems that the same things keep getting tossed. I know that there is the training and it is a great tool to help weed out exactly what severe is, but maybe a module that focuses on better reporting?
As a back on focus, last year my county was one of two in the ARX area that did not have any activations for spotting, that severely lowers my reporting cred, however locally I am one of the few that does go out and reports and usually submit over the local repeater, but being the Skywarn Director I'm trying to encourage a way for folks to track where everyone is and like SN for that feature so I hope you can figure out a way to help folks like me out.
Thanks for a great system, and I think this will help eventually once the few problems are ironed out.

 

[Tyler Allison]

Our training guys are working on an updated module or two specifically on reporting quality.

 

[Greg Stumpf]

Edit: I just looked at my SN account. I got a red flag and a yellow flag for the May 10 tornado near Medford, OK. I wrote this while driving East on SR 11 with a possible rain wrapped tornado to my left....(deletia)

Wes, we really appreciate the reports you provide via SN, but I do have to make a comment here which applies to all users, so I'm not just singling you out...

If a SN observer is *driving*, and can't pull over to type in the report, nor has a passenger who can type in the report, then don't use SN to make the report. You are putting yourself (and other drivers) in danger with this kind of multi-tasking. If you absolutely feel obligated to be able to make spotter reports all the time, even when chasing solo and in the act of driving, employ a redundant alternative means, like a bluetooth phone (as you suggested) or the old standard, a ham radio. Then, the folks you relay the report to can ask you any additional qualifying questions to be sure your report is complete enough to base warning decisions on, and you can a hand on the wheel and your eyes on the road.

 

[Scott Weberpal]

There are three categories:

- High Quality / Perfect Report [ green flag ]
- Acceptable Quality [ yellow flag ]
- Low Quality [ red flag ]

Question about this. According to my "ranking" I have submitted 8 "acceptable" reports and 1 "perfect" report yet when I go back and look at my reports, 8 out of the 9 are all green flags (one yellow flag on one report when I accidentally clicked non-rotating instead of rotating wall cloud). Considering all the other reports have all green flags, shouldn't they be "perfect" reports, while the one with the yellow should be "acceptable"? If that is the case, I should have 8 perfect and 1 acceptable, instead of 1 perfect and 8 acceptable as it is listed now.

 

[John Wetter]

If that is the case, I should have 8 perfect and 1 acceptable, instead of 1 perfect and 8 acceptable as it is listed now.

Were those all this year Scott? The data used for ranking started this past winter.