Homebrewers Association | AHA Forum

General Category => General Homebrew Discussion => Topic started by: S. cerevisiae on March 30, 2014, 04:11:28 AM

Title: Consensus while judging?
Post by: S. cerevisiae on March 30, 2014, 04:11:28 AM
For those who judge competitions, have you ever been on a flight where you could not come to a consensus with another judge?  I am not talking about adjusting a score up or down a few points.  I am talking about a score delta that is so large that the judges have to agree to disagree because no consensus score can be reached.
Title: Re: Consensus while judging?
Post by: gmac on March 30, 2014, 04:26:44 AM
Yes, we brought in another judge from another table to break the tie. He was wrong too.
Title: Re: Consensus while judging?
Post by: dmtaylor on March 30, 2014, 10:51:41 AM
Yes, this occurs when one or more bullheaded judges don't know what the hell they are talking about.  Happens a lot, unfortunately.  I am VERY much AGAINST competition organizers FORCING judges to compromise to come within a certain number of points of one another.  Why not allow each individual to speak for himself/herself!?  Taste and desire is a little subjective.  No one on Earth should be allowed to dictate to another person that what they perceive as desirable or undesirable is wrong.  Yes we have style guidelines for a reason.  No they are not perfect or all-encompassing.  And I am sorry, but I have seen Master judges make as many mistakes as Recognized judges.  We are all human and we should all be entitled to our own opinions.  Compromise drives me crazy.  Leave me alone to do my job the best way I know how based on my own experience, and let the entrant be the judge of the judges as to which one is on mark and which is on base (if any).
Title: Re: Consensus while judging?
Post by: klickitat jim on March 30, 2014, 11:22:19 AM
It must really be a hoot to organize one of these things.
Title: Re: Consensus while judging?
Post by: udubdawg on March 30, 2014, 11:38:01 AM
disagree on which should win a gold medal to the point that a third judge has to come in?  sure.

waaay apart on score and neither can convince the other to budge?  this is where I'd expect myself and the other know-it-alls to come in and do some educatin'....but I've never seen it happen.  Sometimes it has happened when a new judge didn't understand/didn't like the style but always they admitted their score was too low or high.

Title: Re: Consensus while judging?
Post by: klickitat jim on March 30, 2014, 11:51:28 AM
So, the lowest is 13. The highest is 49. That's only 36 points between hell and nirvana. How can they not be within 7 points of each other given the narrow margin?

I guess I have a lot more to learn than I thought.
Title: Re: Consensus while judging?
Post by: mabrungard on March 30, 2014, 12:02:36 PM
I had a similar situation at a recent contest. I scored the beer at 41 and my judging partner scored it at 29...didn't like the hop character. I'm not one to say I'm infallible, so I did drop my score and they compromised a bit too. That beer was eventually pushed to mini BOS and won the category.

An important thing is to make sure that beers that might be good enough to push, get their chance to shine in another forum. When you have large contests with multiple flights, there is a greater chance that other palates will have the opportunity to judge it. The beers in question, just need the opportunity!
Title: Re: Consensus while judging?
Post by: MDixon on March 31, 2014, 12:02:04 PM
I was judging ciders with a brand new guy who had zero experience. He thought everything was wonderful. In the real world he was correct, but we have guidelines. One one beer he refused to budge and I wasn't changing. For some reason I remember it as him being low and we were 10 points apart. I rejected his score entirely and awarded my score. Later in the flight he got with the program, but we did not revisit that cider. I cannot remember if it placed or not in the flight, but I did flip the score sheet and explain what happened to the entrant.

FWIW - I like to be within 3 points.
Title: Re: Consensus while judging?
Post by: klickitat jim on March 31, 2014, 12:22:55 PM
Friday will be my first, if they need me. I'm looking forward to the learning experience.
Title: Re: Consensus while judging?
Post by: dmtaylor on March 31, 2014, 01:00:34 PM
May the Force be with you, padawan learner!
Title: Re: Consensus while judging?
Post by: S. cerevisiae on March 31, 2014, 04:01:43 PM
Here's the situation.  The delta between the scores was almost twenty points.  It was one of those beers that people either loved or hated (50% of the non-flight judges who tasted the beer loved it whereas the other fifty percent thought that it should be dumped).  I was going to give the beer a courtesy score of 13 before I saw the other judge's score sheet.  Our comments were so different that it made me believe that we must have tasted different beers.  I bumped my score up to 29, but there was no way that I was going to give a seriously flawed beer a forty.  The other judge would not budge.  The head judge was clearly uncomfortable judging the category.  He did not have an opinion one way or the other, so he adjusted his score up to move the beer on.   I finally reached the point were I told the head judge to throw out my score because there was no way that I was going to give the beer a score anywhere near forty. 


Title: Re: Consensus while judging?
Post by: a10t2 on March 31, 2014, 05:25:48 PM
I think the only fair thing to do in that situation is to bring in a third judge (or even a second pair of judges if they're available). Otherwise there's no real way to know if a flaw that one judge noted, or a positive element that one didn't, is truly there.

It definitely happens. I entered two beers in the NHC last year that - reading through the pencil erasing - had scoring deltas of 10 and 13. Clearly something was going on between those two judges that goes beyond a minor difference in perception. If one judge calls a beer too malty, and the other says it's too dry, changing the scores to bring them closer together doesn't do anything to get me valid feedback on the beer.
Title: Re: Consensus while judging?
Post by: dmtaylor on March 31, 2014, 05:56:30 PM
As an entrant, I always treat the erased pencil marks as my true scores.  I sometimes wish that the judges wouldn't erase so hard so that I could actually read their original scores.
Title: Re: Consensus while judging?
Post by: udubdawg on March 31, 2014, 06:23:32 PM
Here's the situation.  The delta between the scores was almost twenty points.  It was one of those beers that people either loved or hated (50% of the non-flight judges who tasted the beer loved it whereas the other fifty percent thought that it should be dumped).  I was going to give the beer a courtesy score of 13 before I saw the other judge's score sheet.  Our comments were so different that it made me believe that we must have tasted different beers.  I bumped my score up to 29, but there was no way that I was going to give a seriously flawed beer a forty.  The other judge would not budge.  The head judge was clearly uncomfortable judging the category.  He did not have an opinion one way or the other, so he adjusted his score up to move the beer on.   I finally reached the point were I told the head judge to throw out my score because there was no way that I was going to give the beer a score anywhere near forty.

I'm seeing a lot of "loved it" and not "thought it fit the style really well"
I would like to believe that is not a problem BJCP judges have, but I'd settle for finding out it is rare.

perhaps you could give us a bit more on the style in question and what was so poor about it that made you consider a 13, and eventually 29?



Title: Re: Consensus while judging?
Post by: AmandaK on March 31, 2014, 06:47:12 PM
An important thing is to make sure that beers that might be good enough to push, get their chance to shine in another forum. When you have large contests with multiple flights, there is a greater chance that other palates will have the opportunity to judge it. The beers in question, just need the opportunity!

This is really the most important thing. Ensuring that a medal-worthy beer gets a chance at medaling is more important than worrying about scores.

Usually, if the guy I'm judging with cannot see eye-to-eye with me, I will come down on score with the caveat that the beer will be pushed to mini-BOS. I explain to them that the worst case for that situation is that if the beer is really as bad as they say it is (e.g. a 29 for 'subdued hop aroma' in an APA), then the beer will be kicked immediately. No harm, no foul. But if the beer was as good as I think it is, it should place in mini-BOS. I'm usually correct in pushing the "questionable" beer to mini-BOS (in that they usually medal at that point), but I've been wrong before and seen a beer I fought for get kicked quickly. Better to err on the safe side though!
Title: Re: Consensus while judging?
Post by: hoser on March 31, 2014, 07:00:00 PM
An important thing is to make sure that beers that might be good enough to push, get their chance to shine in another forum. When you have large contests with multiple flights, there is a greater chance that other palates will have the opportunity to judge it. The beers in question, just need the opportunity!

This is really the most important thing. Ensuring that a medal-worthy beer gets a chance at medaling is more important than worrying about scores.

Usually, if the guy I'm judging with cannot see eye-to-eye with me, I will come down on score with the caveat that the beer will be pushed to mini-BOS. I explain to them that the worst case for that situation is that if the beer is really as bad as they say it is (e.g. a 29 for 'subdued hop aroma' in an APA), then the beer will be kicked immediately. No harm, no foul. But if the beer was as good as I think it is, it should place in mini-BOS. I'm usually correct in pushing the "questionable" beer to mini-BOS (in that they usually medal at that point), but I've been wrong before and seen a beer I fought for get kicked quickly. Better to err on the safe side though!

Agree completely with Martin and Amanda! I tend to error on the side of caution if I am not sure.  Had a somewhat similar situation this weekend.  Thought we had a beer that was pretty good so I figured I would give it a chance in many BOS so that a few other palates could taste the beer and decide it's fate.  It placed in the top 3 and will advance to Nationals. 
Title: Re: Consensus while judging?
Post by: chumley on March 31, 2014, 07:43:46 PM
Here's the situation.  The delta between the scores was almost twenty points.  It was one of those beers that people either loved or hated (50% of the non-flight judges who tasted the beer loved it whereas the other fifty percent thought that it should be dumped).  I was going to give the beer a courtesy score of 13 before I saw the other judge's score sheet.  Our comments were so different that it made me believe that we must have tasted different beers.  I bumped my score up to 29, but there was no way that I was going to give a seriously flawed beer a forty.  The other judge would not budge.  The head judge was clearly uncomfortable judging the category.  He did not have an opinion one way or the other, so he adjusted his score up to move the beer on.   I finally reached the point were I told the head judge to throw out my score because there was no way that I was going to give the beer a score anywhere near forty.

If you gave it a 13 and the other guy gave it a 40, then you thought it was flawed and the other guy didn't.  What flaw did you find?

I often have to force myself to be objective when judging IPAs as I really don't like CTZ hops, but a lot of people do, so I have to try to be objective even when I feel like dumping the beer down the drain.
Title: Re: Consensus while judging?
Post by: AmandaK on March 31, 2014, 08:04:54 PM
Here's the situation.  The delta between the scores was almost twenty points.  It was one of those beers that people either loved or hated (50% of the non-flight judges who tasted the beer loved it whereas the other fifty percent thought that it should be dumped).  I was going to give the beer a courtesy score of 13 before I saw the other judge's score sheet.  Our comments were so different that it made me believe that we must have tasted different beers.  I bumped my score up to 29, but there was no way that I was going to give a seriously flawed beer a forty.  The other judge would not budge.  The head judge was clearly uncomfortable judging the category.  He did not have an opinion one way or the other, so he adjusted his score up to move the beer on.   I finally reached the point were I told the head judge to throw out my score because there was no way that I was going to give the beer a score anywhere near forty.

If you gave it a 13 and the other guy gave it a 40, then you thought it was flawed and the other guy didn't.  What flaw did you find?

Also curious. 13 to 40 is a pretty large gap, but I've seen stranger things. :D
Title: Re: Consensus while judging?
Post by: dmtaylor on March 31, 2014, 08:06:24 PM
On my BJCP tasting exam, there was a Belgian dubbel.  Very phenolic, tasted exactly like friggin Carmex.  I believe I even used the term "Carmex" on the tasting sheet.  As such, I scored it relatively low, in the 20s.  It was an otherwise okay dubbel, with the dark fruit flavors, etc., but I just couldn't get past the Carmex.  Meanwhile the Master level proctors all loved it, scored it in the 40s, probably claiming that they loved the rich complex phenols.  = Carmex.  Yuck.  Of course as a result of this disagreement, my exam score was severely impacted, and I remain convinced that I was in the right and they were in the wrong.  I might only be Certified but I don't care what level they were.  I don't want friggin Carmex in any beer that I drink, thank you very much.  No way I would have changed my score upwards for that beer.  After the exam, I also came to find out that many of the other test-takers agreed with me.  If only we could have negotiated with those Master judges, perhaps we could have brought them down.  I wonder how many other takers got screwed that day.

I don't know what the point of all this is, except perhaps to say, taste is subjective, and we should all be entitled to our own opinions.  I have very deep feelings against trying to force anyone to do otherwise.  We can and should compare notes, listen to reason, and adjust scores when appropriate.  However we should also respect those who refuse to budge if they feel very strongly one way or the other.  I think in those cases, we should just let the scoresheets ride as is, and yes, assume that the higher score is the correct one, in fairness to the entrant.
Title: Re: Consensus while judging?
Post by: S. cerevisiae on March 31, 2014, 08:14:50 PM
I'm seeing a lot of "loved it" and not "thought it fit the style really well"
I would like to believe that is not a problem BJCP judges have, but I'd settle for finding out it is rare.

perhaps you could give us a bit more on the style in question and what was so poor about it that made you consider a 13, and eventually 29?

The beer was basically a science experiment that was entered as a specialty beer.   With no claimed "like" beer and no category guidelines to use in judging the beer, I judged the beer based on the ingredients, process, and bugs claimed on the entry form.  The beer had a really harsh middle of the tongue flavor that made it darn near undrinkable for me, which is why I contemplated giving it a 13 (my first score was actually in the low twenties).  I brought my score up because I wanted to reach a consensus. 

As I brew mainly to study the behavior of brewing cultures (I have maintained a culture collection for most of the time that I have brewed), I am familiar with the flavors produced by the bugs claimed in the fermentation.  The harsh off-flavor was not a flavor that is produced by the any of the bugs claimed under normal circumstances.   The flavor was definitely produced by wild non-brewing microflora pickup, which is a flaw that would prevent any beer from scoring in the forties.
 
Title: Re: Consensus while judging?
Post by: AmandaK on March 31, 2014, 08:15:56 PM
Very phenolic, tasted exactly like friggin Carmex.  I believe I even used the term "Carmex" on the tasting sheet.  As such, I scored it relatively low, in the 20s.  It was an otherwise okay dubbel, with the dark fruit flavors, etc., but I just couldn't get past the Carmex.  Meanwhile the Master level proctors all loved it, scored it in the 40s, probably claiming that they loved the rich complex phenols.  = Carmex.  Yuck. 


Yes, taste is subjective, but being able to recognize your limitations and biases is usually the difference between Master exams and the lower scoring exams. (Alongside independent thought, completeness, and thorough descriptive ability, of course.)

For instance, I dislike Fuggles. They are dirt. I could be utterly convinced of dirt=flaw, much like you say "Carmex=flaw". However, I also recognize that Fuggles are totally acceptable in certain categories. So I cannot, and will not, be biased against them in a competition setting.
Title: Re: Consensus while judging?
Post by: Jimmy K on March 31, 2014, 08:34:03 PM
The beer was basically a science experiment that was entered as a specialty beer.   With no claimed "like" beer and no category guidelines to use in judging the beer, I judged the beer based on the ingredients, process, and bugs claimed on the entry form.  The beer had a really harsh middle of the tongue flavor that made it darn near undrinkable for me, which is why I contemplated giving it a 13 (my first score was actually in the low twenties).  I brought my score up because I wanted to reach a consensus. 
Specialty is tough and if there is no reference beer or style listed by the contestant, you're not left with much to judge on besides 'Do I like this'. I imagine this situation comes up most often in the specialty categories. Also, it is up to contestants to adequately prepare judges with specialty ingredients and base styles. I've seen judges who hesitate to give a higher score because they can't adequately decide if the beer was what the brewer intended.
Title: Re: Consensus while judging?
Post by: dmtaylor on March 31, 2014, 10:50:44 PM
taste is subjective, but being able to recognize your limitations and biases is usually the difference between Master exams and the lower scoring exams.

Hmm... good point.  You've given me something to ponder the next time I taste a delicious Belgian ale... that hopefully doesn't taste of Carmex... or Fuggles for that matter!
Title: Re: Consensus while judging?
Post by: udubdawg on March 31, 2014, 11:16:51 PM

The beer was basically a science experiment that was entered as a specialty beer.   With no claimed "like" beer and no category guidelines to use in judging the beer, I judged the beer based on the ingredients, process, and bugs claimed on the entry form.  The beer had a really harsh middle of the tongue flavor that made it darn near undrinkable for me, which is why I contemplated giving it a 13 (my first score was actually in the low twenties).  I brought my score up because I wanted to reach a consensus. 

As I brew mainly to study the behavior of brewing cultures (I have maintained a culture collection for most of the time that I have brewed), I am familiar with the flavors produced by the bugs claimed in the fermentation.  The harsh off-flavor was not a flavor that is produced by the any of the bugs claimed under normal circumstances.   The flavor was definitely produced by wild non-brewing microflora pickup, which is a flaw that would prevent any beer from scoring in the forties.

Ah, fricking category 23...
it's making more sense now.   Yeah you ended up in the most subjective cat.  Science experiments indeed.
I do believe competitions should have very experienced judges in that category, more than just about anywhere else, but perhaps they did.

I don't avoid 23 anymore, but I don't relish it.  I'm hoping the new guidelines will reduce some of the variety; American Wild and Specialty IPA and whatever. 
I am capable of telling why I am/am not impressed with a certain beer's blend of base and specialty information, and from what we've seen I expect you are too.  Curious why the other judge loved it so much, but I guess you really did encounter a fairly rare event early on.  I've never encountered even half of the score differential you indicate.  I actually think you did well with a 29, provided it also included feedback on why it didn't work for you.

cheers--
--Michael
 
Title: Re: Consensus while judging?
Post by: tschmidlin on March 31, 2014, 11:19:44 PM
The beer was basically a science experiment that was entered as a specialty beer.   With no claimed "like" beer and no category guidelines to use in judging the beer, I judged the beer based on the ingredients, process, and bugs claimed on the entry form.  The beer had a really harsh middle of the tongue flavor that made it darn near undrinkable for me, which is why I contemplated giving it a 13 (my first score was actually in the low twenties).  I brought my score up because I wanted to reach a consensus. 
Specialty is tough and if there is no reference beer or style listed by the contestant, you're not left with much to judge on besides 'Do I like this'. I imagine this situation comes up most often in the specialty categories. Also, it is up to contestants to adequately prepare judges with specialty ingredients and base styles. I've seen judges who hesitate to give a higher score because they can't adequately decide if the beer was what the brewer intended.
Specialty and experimental are especially tough if you give too much credence to what the brewer intended.  As a judge in this category, I think it is part of your job to decide if what the brewer intended is actually a good idea.  A perfectly executed tomato weizen is still a terrible beer.  If someone brews a cat vomit beer and it tastes like cat vomit, is that a 50 point beer because that is what the brewer intended?  Or is it a 13 because it tastes like cat vomit?
Title: Re: Consensus while judging?
Post by: klickitat jim on March 31, 2014, 11:28:54 PM
Are you a bottom up judge, or do you go piece by piece and tally afterwards?

The phenolic flaw, assuming that's what it was, seems to have a sliding scale looking at the scoring guide. Very major, like undrinkable (ie gag reflex) then 13, but if it's very minor maybe all the way up to 44. I guess this is why the need for consensus flexibility.

I hope to be the kind of judge that says "this is what I think, but I can be wrong, just tell me where I'm wrong so I can learn"
Title: Re: Consensus while judging?
Post by: garc_mall on March 31, 2014, 11:35:56 PM
The beer was basically a science experiment that was entered as a specialty beer.   With no claimed "like" beer and no category guidelines to use in judging the beer, I judged the beer based on the ingredients, process, and bugs claimed on the entry form.  The beer had a really harsh middle of the tongue flavor that made it darn near undrinkable for me, which is why I contemplated giving it a 13 (my first score was actually in the low twenties).  I brought my score up because I wanted to reach a consensus. 
Specialty is tough and if there is no reference beer or style listed by the contestant, you're not left with much to judge on besides 'Do I like this'. I imagine this situation comes up most often in the specialty categories. Also, it is up to contestants to adequately prepare judges with specialty ingredients and base styles. I've seen judges who hesitate to give a higher score because they can't adequately decide if the beer was what the brewer intended.
Specialty and experimental are especially tough if you give too much credence to what the brewer intended.  As a judge in this category, I think it is part of your job to decide if what the brewer intended is actually a good idea.  A perfectly executed tomato weizen is still a terrible beer.  If someone brews a cat vomit beer and it tastes like cat vomit, is that a 50 point beer because that is what the brewer intended?  Or is it a 13 because it tastes like cat vomit?

The tomatoweizen makes its return!
Title: Re: Consensus while judging?
Post by: tschmidlin on April 01, 2014, 12:08:47 AM
Are you a bottom up judge, or do you go piece by piece and tally afterwards?
I generally go piece by piece and see how it adds up, then decide if that score makes sense for that beer.  Unless it is terrible, then I try to figure out how to make it add up to 13, I never just write a 13 at the bottom and leave it at that.

The tomatoweizen makes its return!
I hated that beer so very very much. :)
Title: Re: Consensus while judging?
Post by: S. cerevisiae on April 01, 2014, 12:17:33 AM
If someone brews a cat vomit beer and it tastes like cat vomit, is that a 50 point beer because that is what the brewer intended?  Or is it a 13 because it tastes like cat vomit?

I have encountered bottles of lambic that had above threshold levels of butyric acid, which gave the beer human vomit notes.  However, I have never encountered cat vomit beer.  ;D
Title: Re: Consensus while judging?
Post by: S. cerevisiae on April 01, 2014, 12:23:35 AM
I hated that beer so very very much. :)

Are you serious?  I thought that you were kidding.  Who would brew a tomatoweizen?  Better yet, who would enter it in a contest?
Title: Re: Consensus while judging?
Post by: MDixon on April 01, 2014, 01:05:58 AM
The bottom line is they had two judges on the flight who had no business being there. New judges should not do specialty and neither should those who cannot make a decision.

13 is a very low score...think undrinkable wretching at the flavor making you want to blow chunks. I rarely go down there because I can award points other places. I'm guessing the 13 was too low. Sounds like the other score was too high and the middle ground is more appropriate.
Title: Re: Consensus while judging?
Post by: S. cerevisiae on April 01, 2014, 03:58:03 AM
I only considered giving the beer a thirteen because I could barely drink it. I never actually wrote thirteen on the score sheet.  My first score was in the low twenties, which I thought was generous.   I was completely flabbergasted when I saw that the other judge had written "Fantastic!" on his score sheet (his score was in the low forties).   
Title: Re: Consensus while judging?
Post by: tschmidlin on April 01, 2014, 05:02:45 AM
I hated that beer so very very much. :)

Are you serious?  I thought that you were kidding.  Who would brew a tomatoweizen?  Better yet, who would enter it in a contest?
Unfortunately it wasn't for a contest, a guy brought a keg to a meeting so I had to drink it in front of him and tell him what I thought of it.   :(

I agree 13 is really low and I can usually score a beer higher than that.  But I've had some that 13 was truly a courtesy, even the appearance was a 1/3 at best.  And there have been some that I refused to taste, and it's hard to score taste/mouthfeel when the smell is so offensive you don't want to put it in your mouth.
Title: Re: Consensus while judging?
Post by: klickitat jim on April 01, 2014, 09:32:36 AM
The new guide needs a Dare section. Ordinary Dare, Extra Special Dare, and Double Dog Dare.
Title: Re: Consensus while judging?
Post by: HoosierBrew on April 01, 2014, 11:59:19 AM
I hated that beer so very very much. :)

Are you serious?  I thought that you were kidding.  Who would brew a tomatoweizen?  Better yet, who would enter it in a contest?

Wow, I thought it was a joke too. There's pushing the brewing envelope, and then there's just brewing s^%t.
Title: Re: Consensus while judging?
Post by: Jimmy K on April 01, 2014, 12:45:07 PM
I hated that beer so very very much. :)

Are you serious?  I thought that you were kidding.  Who would brew a tomatoweizen?  Better yet, who would enter it in a contest?

Wow, I thought it was a joke too. There's pushing the brewing envelope, and then there's just brewing s^%t.
After black garlic stout, I think nothing is a joke.
Title: Re: Consensus while judging?
Post by: reverseapachemaster on April 01, 2014, 02:32:34 PM
I judged a beer last month for a competition and it was just awful. I gave it a very generous 19 because the competition didn't want to score any lower unless there was a good reason and apparently horrendous taste just wasn't good enough. It was like somebody really screwed up their first beer and decided to enter it in a competition anyway. At least my judging partner gave it a similar score.
Title: Re: Consensus while judging?
Post by: denny on April 01, 2014, 04:02:47 PM
As an entrant, I always treat the erased pencil marks as my true scores.  I sometimes wish that the judges wouldn't erase so hard so that I could actually read their original scores.

I think that's a big mistake.  I know that most of the time when I change a score, it's because my initial score was reached too quickly and after further consideration and tasting I have changed my mind.
Title: Re: Consensus while judging?
Post by: denny on April 01, 2014, 04:04:50 PM
I agree 13 is really low and I can usually score a beer higher than that.  But I've had some that 13 was truly a courtesy, even the appearance was a 1/3 at best.  And there have been some that I refused to taste, and it's hard to score taste/mouthfeel when the smell is so offensive you don't want to put it in your mouth.

John Maier was judging at a comp I was running once.   We had set the low score as 13.  There was a particular beer that he absolutely refused to score higher than 5.
Title: Re: Consensus while judging?
Post by: gmac on April 01, 2014, 04:07:42 PM
As an entrant, I always treat the erased pencil marks as my true scores.  I sometimes wish that the judges wouldn't erase so hard so that I could actually read their original scores.

I think that's a big mistake.  I know that most of the time when I change a score, it's because my initial score was reached too quickly and after further consideration and tasting I have changed my mind.

I agree, I often find myself going back and adjusting beers that were scored early in the flight or late in the flight because I was being too critical early on.  My final score is probably more real and maybe more useful than the first impression.
Title: Re: Consensus while judging?
Post by: AmandaK on April 01, 2014, 04:42:23 PM
As an entrant, I always treat the erased pencil marks as my true scores.  I sometimes wish that the judges wouldn't erase so hard so that I could actually read their original scores.

I think that's a big mistake.  I know that most of the time when I change a score, it's because my initial score was reached too quickly and after further consideration and tasting I have changed my mind.

I agree, I often find myself going back and adjusting beers that were scored early in the flight or late in the flight because I was being too critical early on.  My final score is probably more real and maybe more useful than the first impression.

+3

Plus, as an entrant I'm less worried about the scores and more worried about the feedback anyway. A 42 isn't going to make my beer better, but useful feedback will.
Title: Re: Consensus while judging?
Post by: jeffy on April 01, 2014, 05:02:13 PM
At the NHC in Orlando I was alerted to a pair of score sheets that gave a beer a score of 12 (I think) with very few comments on the sheet other than "some oxidized notes."  The two judges were nationally known and respected.  As competition organizer I asked one of them if there was anything else about it that made it score so low.  When the answer was "no" I got another BJCP National and we rejudged the beer so we wouldn't have to explain the bad score sheets to the entrant.  I think we gave it a 33.  Good beer with a minor flaw.
I don't mind seeing low scores, but there had better be a lot of writing to explain it.
Title: Re: Consensus while judging?
Post by: udubdawg on April 01, 2014, 05:13:11 PM
At the NHC in Orlando I was alerted to a pair of score sheets that gave a beer a score of 12 (I think) with very few comments on the sheet other than "some oxidized notes."  The two judges were nationally known and respected.  As competition organizer I asked one of them if there was anything else about it that made it score so low.  When the answer was "no" I got another BJCP National and we rejudged the beer so we wouldn't have to explain the bad score sheets to the entrant.  I think we gave it a 33.  Good beer with a minor flaw.
I don't mind seeing low scores, but there had better be a lot of writing to explain it.

Now that's a good competition organizer. 
"I'm in charge, this isn't right, and I'm not putting my name on something where an entrant is treated like this"
Title: Re: Consensus while judging?
Post by: dixonmike20 on April 01, 2014, 06:24:51 PM
That large a disparity would indicate one of the judges is wrong on style or being able to perceive flavors.

I would expected flavor perception would be effected if they've been judging a lot of beers. The palate gets bombarded.

Also, I've seen experienced judges clearly not know the style guideline, but act like they know it all. It's not hard, they can always look at the guidelines to be sure.
 
It could also be a matter of exposure to the style. Knowing the guideline is different than having experience tasting commercial examples. For instance, many Americans, even judges, would not recognize an English IPA as an IPA, even when accounting for a lower IBU. If they don't have a frame of reference, they simply don't know what its supposed to taste like. Taste as many English IPAs available here or spend some time in the UK tasting different IPAs and you'll understand. The same goes for any style.


Sent from my iPhone using Tapatalk (http://tapatalk.com/m?id=1)
Title: Re: Consensus while judging?
Post by: AmandaK on April 01, 2014, 06:27:30 PM
At the NHC in Orlando I was alerted to a pair of score sheets that gave a beer a score of 12 (I think) with very few comments on the sheet other than "some oxidized notes."  The two judges were nationally known and respected.  As competition organizer I asked one of them if there was anything else about it that made it score so low.  When the answer was "no" I got another BJCP National and we rejudged the beer so we wouldn't have to explain the bad score sheets to the entrant.  I think we gave it a 33.  Good beer with a minor flaw.
I don't mind seeing low scores, but there had better be a lot of writing to explain it.

Now that's a good competition organizer. 
"I'm in charge, this isn't right, and I'm not putting my name on something where an entrant is treated like this"

Exactly my thoughts! Good work Jeffy.
Title: Re: Consensus while judging?
Post by: dixonmike20 on April 01, 2014, 06:31:44 PM

At the NHC in Orlando I was alerted to a pair of score sheets that gave a beer a score of 12 (I think) with very few comments on the sheet other than "some oxidized notes."  The two judges were nationally known and respected.  As competition organizer I asked one of them if there was anything else about it that made it score so low.  When the answer was "no" I got another BJCP National and we rejudged the beer so we wouldn't have to explain the bad score sheets to the entrant.  I think we gave it a 33.  Good beer with a minor flaw.
I don't mind seeing low scores, but there had better be a lot of writing to explain it.

Now that's a good competition organizer. 
"I'm in charge, this isn't right, and I'm not putting my name on something where an entrant is treated like this"

Exactly my thoughts! Good work Jeffy.

+1

There's no excuse for that kind of "judging". Give feedback or don't bother judging.


Sent from my iPhone using Tapatalk (http://tapatalk.com/m?id=1)
Title: Re: Consensus while judging?
Post by: gmac on April 01, 2014, 10:03:06 PM

At the NHC in Orlando I was alerted to a pair of score sheets that gave a beer a score of 12 (I think) with very few comments on the sheet other than "some oxidized notes."  The two judges were nationally known and respected.  As competition organizer I asked one of them if there was anything else about it that made it score so low.  When the answer was "no" I got another BJCP National and we rejudged the beer so we wouldn't have to explain the bad score sheets to the entrant.  I think we gave it a 33.  Good beer with a minor flaw.
I don't mind seeing low scores, but there had better be a lot of writing to explain it.

Now that's a good competition organizer. 
"I'm in charge, this isn't right, and I'm not putting my name on something where an entrant is treated like this"

Exactly my thoughts! Good work Jeffy.

+1

There's no excuse for that kind of "judging". Give feedback or don't bother judging.


Sent from my iPhone using Tapatalk (http://tapatalk.com/m?id=1)
That's bull.  I just judged a competition of commercial brewers and there were some surprisingly bad ones for commercial brewers and I highly doubt they really cared what I had to say about making improvements but I still gave it because that's what I'm there for.  Are they going to make dramatic changes based on my amateur comments when they are making a living making beer, doubtful and maybe they just wanted a score to get a medal for marketing purposes but like I said, that's not what I'm there for.
Title: Re: Consensus while judging?
Post by: dixonmike20 on April 01, 2014, 10:10:08 PM
Well, we're discussing homebrew competitions. Homebrewers do want feedback.

If you're too lazy to do it right, don't judge.


Sent from my iPhone using Tapatalk (http://tapatalk.com/m?id=1)
Title: Re: Consensus while judging?
Post by: gmac on April 01, 2014, 10:21:35 PM
Well, we're discussing homebrew competitions. Homebrewers do want feedback.

If you're too lazy to do it right, don't judge.


Sent from my iPhone using Tapatalk (http://tapatalk.com/m?id=1)

Sorry, I was referring to the 12 without comments when I said "That's bull", not your post if thats what you thought.  I'm saying feedback is a must regardless although I thought that was a bit clearer in my comments.
Title: Re: Consensus while judging?
Post by: dixonmike20 on April 01, 2014, 10:22:16 PM
Sorry. Misunderstanding.


Sent from my iPhone using Tapatalk (http://tapatalk.com/m?id=1)
Title: Re: Consensus while judging?
Post by: gmac on April 01, 2014, 10:23:47 PM
No problem.  I was trying to say I'm giving feedback whether they want it or not. 
Title: Re: Consensus while judging?
Post by: dixonmike20 on April 01, 2014, 10:59:55 PM
Gotcha


Sent from my iPhone using Tapatalk (http://tapatalk.com/m?id=1)
Title: Re: Consensus while judging?
Post by: chezteth on April 01, 2014, 11:00:31 PM
I judged an onion stout (not a BJCP comp) that tasted like rotten onions. Apparently the brewer sautéed onions in butter then added it to his beer. I tried my best to look past the rotten onion flavor & aroma and as far as I could tell the base beer was OK. As was said before... Some things should not be used in beer.

Sent from my Nexus 7 using Tapatalk

Title: Re: Consensus while judging?
Post by: YooperBrew on April 02, 2014, 01:56:44 AM
I hated that beer so very very much. :)

Are you serious?  I thought that you were kidding.  Who would brew a tomatoweizen?  Better yet, who would enter it in a contest?

Wow, I thought it was a joke too. There's pushing the brewing envelope, and then there's just brewing s^%t.
After black garlic stout, I think nothing is a joke.

Let's see.  I judged a "wild" stout competition, and had a French Onion Soup stout, Rosemary Stout (tasted like lamb roast), a creme brulee' stout, and a few others than I've permanently blocked from my brain.

Of those, the French Onion Soup Stout was the grossest.  It gushed, it stank of onions, and it tasted amazingly like French onion soup without the cheese. 

Never, ever, think that any beer is too gross or weird to be entered into competition.

Title: Re: Consensus while judging?
Post by: dkfick on April 02, 2014, 02:02:34 AM
Kuhnhenn's makes a creme brulee' stout that is a-maze-ing.
Title: Consensus while judging?
Post by: Stevie on April 02, 2014, 02:12:42 AM
Kuhnhenn's makes a creme brulee' stout that is a-maze-ing.

So does southern tier. It's somewhere around $10 a bomber. Too much for me to risk it.
Title: Re: Consensus while judging?
Post by: dkfick on April 02, 2014, 02:20:19 AM
Kuhnhenn's makes a creme brulee' stout that is a-maze-ing.

So does southern tier. It's somewhere around $10 a bomber. Too much for me to risk it.
I've had that one too, it's quite good as well.  Worth $10 since it's basically 10%... bring a friend lol
Title: Re: Consensus while judging?
Post by: rjberry on April 02, 2014, 02:42:42 AM
Kuhnhenn's makes a creme brulee' stout that is a-maze-ing.

So does southern tier. It's somewhere around $10 a bomber. Too much for me to risk it.

It's well worth the $10.  A little on the sweet side, but fantastic for a nightcap
Title: Re: Consensus while judging?
Post by: tschmidlin on April 02, 2014, 05:39:16 AM
Kuhnhenn's makes a creme brulee' stout that is a-maze-ing.
So does southern tier. It's somewhere around $10 a bomber. Too much for me to risk it.
It's well worth the $10.  A little on the sweet side, but fantastic for a nightcap
Agreed, great beer.  Sweet and strong, so it would be hard to drink 22 oz by yourself, that's why ...
bring a friend
;)
Title: Re: Consensus while judging?
Post by: rjberry on April 02, 2014, 12:48:28 PM
Kuhnhenn's makes a creme brulee' stout that is a-maze-ing.
So does southern tier. It's somewhere around $10 a bomber. Too much for me to risk it.
It's well worth the $10.  A little on the sweet side, but fantastic for a nightcap
Agreed, great beer.  Sweet and strong, so it would be hard to drink 22 oz by yourself, that's why ...
bring a friend
;)

Luckily the wife enjoys beer as much as I do so we go splits on the bombers  ;)

Quite the tangent on a beer scoring thread
Title: Re: Consensus while judging?
Post by: klickitat jim on April 03, 2014, 02:20:06 AM
As an entrant, I always treat the erased pencil marks as my true scores.  I sometimes wish that the judges wouldn't erase so hard so that I could actually read their original scores.

Sometime erasing can be problematic
(http://img.tapatalk.com/d/14/04/03/7u5evype.jpg)
Title: Re: Consensus while judging?
Post by: tschmidlin on April 03, 2014, 04:15:42 AM
As an entrant, I always treat the erased pencil marks as my true scores.  I sometimes wish that the judges wouldn't erase so hard so that I could actually read their original scores.

Sometime erasing can be problematic
(http://img.tapatalk.com/d/14/04/03/7u5evype.jpg)
Ha!  Strange rubbery odor indeed.  It smells like pencil eraser. ;D
Title: Re: Consensus while judging?
Post by: klickitat jim on April 03, 2014, 04:38:10 AM
When I first got interested in passing judgement, I read that judges must use mechanical pencils so their sense of smell isn't disturbed. This concerns me. Pencils have no odor to me unless they are on fire. Maybe it's years of smelling things most people could not imagine. I'm hoping smell ability doesn't have to be that acute. I'm curious if scratch paper, like a note pad, is allowed. Because I like to scribble and write my final thoughts on the score sheet. Rather than schmuck it up or fill the room with hot eraser fumes.
Title: Re: Consensus while judging?
Post by: tschmidlin on April 03, 2014, 06:01:12 AM
Pencils smell like cedar, especially when they are just sharpened.  Try it.

You are allowed to use scratch paper of course, but I would discourage it.  You are trying to get your thoughts down completely and quickly.  You can't spend 20 minutes judging a single beer.
Title: Re: Consensus while judging?
Post by: MDixon on April 03, 2014, 11:28:12 AM
Pencils absolutely have an odor.

You could use scratch paper, but you are slowing down the process. Smell, see, taste, or sense it and then put it on the score sheet. Too many people sit around and think rather than writing what they perceived. If you truly perceive it, you will not be wrong to write that on the score sheet.
Title: Re: Consensus while judging?
Post by: dixonmike20 on April 03, 2014, 11:54:11 AM
^ great name


Sent from my iPhone using Tapatalk (http://tapatalk.com/m?id=1)
Title: Re: Consensus while judging?
Post by: dmtaylor on April 03, 2014, 01:16:10 PM
Sometime erasing can be problematic
(http://img.tapatalk.com/d/14/04/03/7u5evype.jpg)

Looks like this one was completed by a second grader.  I'd pitch it.  Gosh, I'm a hell of a jerk, aren't I.
Title: Re: Consensus while judging?
Post by: klickitat jim on April 03, 2014, 04:37:23 PM
The photo of the score sheet is a joke just in case anyone thought it was real.
Title: Re: Consensus while judging?
Post by: dixonmike20 on April 03, 2014, 04:41:38 PM
I remember getting one that had one word comments on each category. But, the overall comments listed things the beer should not have and every one was verbatim against the style guide. For example, it was a 80/- and he said it should have a big hop aroma.

If I didn't know it was real, I would've thought it was a joke.


Sent from my iPhone using Tapatalk (http://tapatalk.com/m?id=1)
Title: Re: Consensus while judging?
Post by: dmtaylor on April 03, 2014, 07:43:49 PM
Didn't look like a joke to me.  I've seen that level of quality, and worse, from real competitions.
Title: Re: Consensus while judging?
Post by: MDixon on April 04, 2014, 12:40:21 AM
^ great name


Thanks, I've had it for closing in on a half century ;)

Title: Re: Consensus while judging?
Post by: klickitat jim on April 04, 2014, 01:21:26 AM
Didn't look like a joke to me.  I've seen that level of quality, and worse, from real competitions.

That scares me. I made it up to be humorous and took a photo. My goal is to do better than that in reality.
Title: Re: Consensus while judging?
Post by: tschmidlin on April 04, 2014, 06:00:41 AM
Didn't look like a joke to me.  I've seen that level of quality, and worse, from real competitions.

That scares me. I made it up to be humorous and took a photo. My goal is to do better than that in reality.
Yeah, I thought it was a real one too. :)
Title: Re: Consensus while judging?
Post by: klickitat jim on April 04, 2014, 06:19:34 AM
Yikes
Title: Re: Consensus while judging?
Post by: james on April 04, 2014, 07:23:55 AM
Yikes

I've seen worse as well
Title: Re: Consensus while judging?
Post by: Jimmy K on April 04, 2014, 11:36:03 AM
Well played

Sent from my XT1030 using Tapatalk

Title: Re: Consensus while judging?
Post by: jeffy on April 04, 2014, 11:38:57 AM
Didn't look like a joke to me.  I've seen that level of quality, and worse, from real competitions.

That scares me. I made it up to be humorous and took a photo. My goal is to do better than that in reality.
Yeah, I thought it was a real one too. :)

Me, too.  I've seen so many bad score sheets I didn't even give it a second thought.
It took a hint for me to get the rubbery smell joke.
Title: Re: Consensus while judging?
Post by: morticaixavier on April 16, 2014, 07:50:56 PM
Ahh experience.

Just got my sheets back from NHC first round and on one of the ones that didn't make it (pretty sure it would have if not for what I am about to share) I had three sheets in the high 30's with evident erasures that appear to have it in the low 40's and one at 31 that reads like the judge was drinking a different beer. Flaws galore that apparently only that judge could perceive. From cheesy hops to burning alcohol flavor. Also, apparently too many esters for an english barley wine (???)

anyway. This will be the only complaining I do about this but I thought it was cogent to this discussion and my wife won't let me talk about it at home soo....
Title: Re: Consensus while judging?
Post by: a10t2 on April 16, 2014, 09:01:33 PM
Four judges? Showoff. :P
Title: Re: Consensus while judging?
Post by: james on April 16, 2014, 09:13:00 PM
Ahh experience.

Just got my sheets back from NHC first round and on one of the ones that didn't make it (pretty sure it would have if not for what I am about to share) I had three sheets in the high 30's with evident erasures that appear to have it in the low 40's and one at 31 that reads like the judge was drinking a different beer. Flaws galore that apparently only that judge could perceive. From cheesy hops to burning alcohol flavor. Also, apparently too many esters for an english barley wine (???)

anyway. This will be the only complaining I do about this but I thought it was cogent to this discussion and my wife won't let me talk about it at home soo....

I took 1st in strong ale in that region, when I get my sheets back I'll let you know how they look.  Of course with 30 entries I'd hope there were at least 3-4 flights for the category. 

I've judged a dozen strong ales in a flight before and that wasn't fun
Title: Re: Consensus while judging?
Post by: morticaixavier on April 16, 2014, 09:18:41 PM
Ahh experience.

Just got my sheets back from NHC first round and on one of the ones that didn't make it (pretty sure it would have if not for what I am about to share) I had three sheets in the high 30's with evident erasures that appear to have it in the low 40's and one at 31 that reads like the judge was drinking a different beer. Flaws galore that apparently only that judge could perceive. From cheesy hops to burning alcohol flavor. Also, apparently too many esters for an english barley wine (???)

anyway. This will be the only complaining I do about this but I thought it was cogent to this discussion and my wife won't let me talk about it at home soo....

I took 1st in strong ale in that region, when I get my sheets back I'll let you know how they look.  Of course with 30 entries I'd hope there were at least 3-4 flights for the category. 

I've judged a dozen strong ales in a flight before and that wasn't fun

Congrates! mine was actually in cat 23 because it had maple syrup in it. I imagine judgeing 28 cat 23 beers is even more overwhelming than strong ales though.
Title: Re: Consensus while judging?
Post by: jeffy on April 16, 2014, 09:23:04 PM
Ahh experience.

Just got my sheets back from NHC first round and on one of the ones that didn't make it (pretty sure it would have if not for what I am about to share) I had three sheets in the high 30's with evident erasures that appear to have it in the low 40's and one at 31 that reads like the judge was drinking a different beer. Flaws galore that apparently only that judge could perceive. From cheesy hops to burning alcohol flavor. Also, apparently too many esters for an english barley wine (???)

anyway. This will be the only complaining I do about this but I thought it was cogent to this discussion and my wife won't let me talk about it at home soo....

I took 1st in strong ale in that region, when I get my sheets back I'll let you know how they look.  Of course with 30 entries I'd hope there were at least 3-4 flights for the category. 

I've judged a dozen strong ales in a flight before and that wasn't fun
I've also judged that many in a flight and on the contrary it was great fun.