Author Topic: Judge experience  (Read 944 times)

Offline Iliff Ave Brewhouse

  • Official Poobah of No Life.
  • *
  • Posts: 3774
Judge experience
« on: August 14, 2018, 05:51:00 PM »
Lately I seem to notice a trend in comps I enter where I have two judges of different experience levels. The less experienced judges seem to give me scores that are about 4-6 points lower than the more experienced judges. Has any one else experienced this? What would be the explanation?

The only thing that makes sense is that the less experienced judges are looking for more flaws to prove they know what they are doing. It's almost they are looking for the good parts of a beer secondarily or as an afterthought. It's been very frustrating for me so I pretty much ignore those scores even though the two scores get averaged for my final.
« Last Edit: August 14, 2018, 05:57:00 PM by goschman »
On Tap/Bottled: Dunkel, Lozacca Lager, Session Lager, Helles Bock, Cherry American Strong

Fermenting: Mojito Cider, Citralaxy IPA
Up Next: Golden Stout, G Pils, Sticke Alt

Offline jeffy

  • Official Poobah of No Life.
  • *
  • Posts: 3687
  • Tampa, Fl
Re: Judge experience
« Reply #1 on: August 14, 2018, 06:40:38 PM »
I have been observing the opposite lately.  As the more experienced judge I usually score a little less than the newer guy I am paired with.  We generally have a lot of the same comments and criticisms but I don't feel they are adhering to the little boxes at the bottom left that tell where a beer with such and such number of flaws should fit.  For example we may both notice flaws with carbonation and attenuation, but he/she still scores it mid-30's, while I am under 30. 
I think newer judges tend to be more lenient.
Jeff Gladish, Tampa (989.3, 175.1 Apparent Rennarian)
Homebrewing since 1990
AHA member since 1991, now a lifetime member
BJCP judge since 1995

Offline dmtaylor

  • Official Poobah of No Life.
  • *
  • Posts: 3839
    • Manty Malters - Meet the Malters! - Dave Taylor
Re: Judge experience
« Reply #2 on: August 14, 2018, 06:44:39 PM »
Most typically I find that the less experienced judges very obviously copy some of the results from higher ranked judges, and it drives me up the wall because I would prefer that each judge give me THEIR OWN impressions.  I don't care at all about consensus, whether as an entrant or as a judge myself.  If I got totally different results from two different judges, I'd be both shocked AND pleased.

I understand what you are talking about though too.  I agree that less experienced judges can have a tendency to try to show off their knowledge, even to the point of their comments becoming absurd.  Some judges detect diacetyl in every single beer they taste, whether it's actually there or not.  Some judges will say every beer is oxidized and "probably better when it was fresher".  On the other hand I also have seen a Master level judge who claimed he was able to detect astringency in the aroma.  Yeah, right.  Scoresheets like this just go straight into the recycle bin for me.

It is for many of these reasons, in part, that I enter very few competitions anymore these days.  I enjoy judging, not saying I'm perfect at it either though, but I'd really rather judge than enter.  I think I entered one a couple years ago.  Some judges do an outstanding job, but they are few and far between IMO.  One thing I can promise to anyone who receives a scoresheet from yours truly -- I actually wrote down what *I* tasted, and I didn't give a damn about what the guy sitting next to me wrote on his sheet.  What you see is what I myself actually perceived.  I sure wish more people would judge that way.

Cheers.
Dave

The world will become a much more pleasant place to live when each and every one of us realizes that we are all idiots.

Offline James K

  • Brewer
  • ****
  • Posts: 441
  • Flagstaff, AZ
Re: Judge experience
« Reply #3 on: August 14, 2018, 09:32:26 PM »
I actually wrote down what *I* tasted, and I didn't give a damn about what the guy sitting next to me wrote on his sheet.  What you see is what I myself actually perceived.  I sure wish more people would judge that way.

Cheers.
Im not the most experienced judge but I agree with this 100% and perhaps it’s the format of how my club has taught me to judge. When we judge anything, club meeting or a competition, we don’t even talk to one another while tasting. We all drink our little 4oz taster, mostly we don’t even need that much, and then we score, referring back to the beer if needed. After everyone’s card has been filled out, we share our points and then say why or why not. If someone needs to go up or down/up in score we adjust but no one is taking the information off of another judges card.

I recently judged a mead competition and I don’t know a whole lot about mead. I just followed the guidelines though and when one of the meads we moved forward made it to the finals everyone agreed that it should have been there.

I don’t see how you can copy someone’s score card though when everyone’s pallet is different. I’m hortible with detecting some things. And I’m great at detecting and talking about other things. You have to have a balance, otherwise it’s all subjective. IMO.
« Last Edit: August 14, 2018, 09:34:16 PM by James K »
Vice President of Flagstaff Mountain-Top Mashers
2017 Homebrewer of the year
"One mouth doesn't taste the beer."

Offline Iliff Ave Brewhouse

  • Official Poobah of No Life.
  • *
  • Posts: 3774
Re: Judge experience
« Reply #4 on: August 14, 2018, 09:46:21 PM »
Thanks all. The only reason I enter comps is to get feedback. The most frustrating part lately is even decent scores get more negative feedback than positive.
On Tap/Bottled: Dunkel, Lozacca Lager, Session Lager, Helles Bock, Cherry American Strong

Fermenting: Mojito Cider, Citralaxy IPA
Up Next: Golden Stout, G Pils, Sticke Alt

Offline dls5492

  • Brewer
  • ****
  • Posts: 367
Re: Judge experience
« Reply #5 on: August 14, 2018, 10:51:44 PM »
I like entering the same beer in a couple of competitions. What I look for is if the different judges are making similar comments. I look for a pattern.
I don't mind negative comments. Unless I get a 50, I want idea(s) to improve my beer. I appreciate all the work judges put in and thank every one of you!
David S.
Cedar Falls, IA
Club: Cedar River Association of Zymurgy Enthusiasts (CRAZE)

And the Lord is the Spirit; and where the Spirit of the Lord is, there is liberty. II Cor. 3:17

Offline ethinson

  • Brewmaster
  • *****
  • Posts: 561
  • Why is the beer always gone?
    • River Pirate Brewing Co.
Re: Judge experience
« Reply #6 on: August 15, 2018, 12:09:17 PM »
Thanks all. The only reason I enter comps is to get feedback. The most frustrating part lately is even decent scores get more negative feedback than positive.

If they are listing flaws and not listing ways to correct it then I would be worried, but if they are offering possible solutions than that's a good thing.  In theory, a judge should list everything they smell, see and taste.  So they might write "Light DMS" for an adjunct lager.  That's OK to style so they won't list a reason to fix that.  So yes, the more comments will be about flaws that need fixing so there will be more "negative" than positive.  But on the flip side of that coin, not everything listed is a flaw.  That took me a long time to wrap my head around both as a judge and as a competitor.  I'd get scoresheets back with some flavor listed and be like "That's supposed to be in there!!" assuming they had docked it for it.  Probably not. 

As far as the point difference, the BJCP guide says for judges to be within 7 points, so 4-6 is not bad.  Oddly enough in my experience, novice judges tend to score higher than experienced judges, but everyone is different.  I think your theory is valid though of a less experienced judge thinking "Ok, this is a point off and this is a point off and this is a point off" and shaving a lot of points off for minor flaws.
SE Portland - AKA Beervana
Captain and Chief Deck Swabber - River Pirate Brewing Co.
Certified BJCP Beer Judge
2015 Oregon Brew Crew Member of the Year

Offline Iliff Ave Brewhouse

  • Official Poobah of No Life.
  • *
  • Posts: 3774
Re: Judge experience
« Reply #7 on: August 15, 2018, 01:43:17 PM »
Good to hear others feedback. I am in no way disparaging any judges but just thought it was an interesting occurrence. It is likely just the pair of judges I got. It seems that as I have become a better brewer, my scores have gotten worse which is a bit frustrating. I don't think I will truly ever understand competitions but that's okay.
On Tap/Bottled: Dunkel, Lozacca Lager, Session Lager, Helles Bock, Cherry American Strong

Fermenting: Mojito Cider, Citralaxy IPA
Up Next: Golden Stout, G Pils, Sticke Alt

Offline dmtaylor

  • Official Poobah of No Life.
  • *
  • Posts: 3839
    • Manty Malters - Meet the Malters! - Dave Taylor
Re: Judge experience
« Reply #8 on: August 15, 2018, 02:05:04 PM »
I am in no way disparaging any judges

But *I* am.   ;D

but just thought it was an interesting occurrence. It is likely just the pair of judges I got. It seems that as I have become a better brewer, my scores have gotten worse which is a bit frustrating. I don't think I will truly ever understand competitions but that's okay.

I've said this a hundred times, so I'll say it again:  The best way to handle all the inherent flaws with competitions is either: A) to give up and never enter at all, or B) to enter each beer into AT LEAST 3 or 4 competitions, then sift through all 6-10 scoresheets, throw out the 60% worthless scoresheets, and keep just the 40% that have any useful comments.  Yes, that's right.  Throw the bad ones straight into the recycle bin and forget that they ever happened.  This might sound funny to some but I swear I am dead serious.  That's how to resolve the issue, because BJCP can't fix it on the other end, never can, never will, impossible.

Cheers.
Dave

The world will become a much more pleasant place to live when each and every one of us realizes that we are all idiots.

Offline reverseapachemaster

  • Official Poobah of No Life.
  • *
  • Posts: 3328
    • Brain Sparging on Brewing
Re: Judge experience
« Reply #9 on: August 15, 2018, 04:14:57 PM »
You can get that difference in scores just by whether the judge scores by looking to take points away or build points up. In each situation a judge who thinks flavor could be a six or seven is going to round one way or the other. Have a judge with each paradigm at the table do that on two or three scoring categories and you'll consistently have a four to six point spread.
Heck yeah I blog about homebrewing: Brain Sparging on Brewing

Offline reverseapachemaster

  • Official Poobah of No Life.
  • *
  • Posts: 3328
    • Brain Sparging on Brewing
Re: Judge experience
« Reply #10 on: August 15, 2018, 04:16:46 PM »
Good to hear others feedback. I am in no way disparaging any judges but just thought it was an interesting occurrence. It is likely just the pair of judges I got. It seems that as I have become a better brewer, my scores have gotten worse which is a bit frustrating. I don't think I will truly ever understand competitions but that's okay.

Easily possible that as your beers improve they are improving towards what you like to drink rather than what scores well in a competition. An IPA with citra and mosaic is more likely to score well over an IPA with the 4Cs even if the latter better satiates the guidelines, just because judges tend to vote in favor of what they like and what they like is often whatever is popular at the moment.
Heck yeah I blog about homebrewing: Brain Sparging on Brewing

Offline denny

  • Administrator
  • Retired with too much time on my hands
  • *****
  • Posts: 21723
  • Noti OR [1991.4, 287.6deg] AR
    • Dennybrew
Re: Judge experience
« Reply #11 on: August 15, 2018, 06:02:19 PM »
Based on my experience, any generalization about judging is bound to be wrong.....Dave....;)
Life begins at 60.....1.060, that is!

www.dennybrew.com

The best, sharpest, funniest, weirdest and most knowledgable minds in home brewing contribute on the AHA forum. - Alewyfe

"The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts." - Bertrand Russell

Offline dmtaylor

  • Official Poobah of No Life.
  • *
  • Posts: 3839
    • Manty Malters - Meet the Malters! - Dave Taylor
Re: Judge experience
« Reply #12 on: August 15, 2018, 06:42:57 PM »
Based on my experience, any generalization about judging is bound to be wrong.....Dave....;)

I won't argue with that.  We're all over the dang place.  Even the same judge on two different days will often (usually?) give two different results.
Dave

The world will become a much more pleasant place to live when each and every one of us realizes that we are all idiots.