Author Topic: Bias in BJCP judging?  (Read 4751 times)

Offline S. cerevisiae

  • Senior Brewmaster
  • ******
  • Posts: 1696
  • deus ex machina
    • View Profile
Re: Bias in BJCP judging?
« Reply #15 on: March 02, 2014, 09:09:30 AM »
Outside of personal bias creeping into decisions, I believe that the BJCP program does an excellent job of teaching style parameters.  All of the national and most of the recognized judges with whom I interacted at the competition knew style parameters very well.  In my humble opinion, the area in which the program could use some improvement is teaching the accurate identification of fermentation metabolic byproducts, especially esters.  Beer after beer, any hint of apple was flagged as suffering from incomplete fermentation.  However, the presence of apple does not always mean that a beer is suffering from incomplete fermentation.  The type of apple matters.  Green apple is acetaldehyde.   Ripe red apple is ethyl hexanoate.  Acetaldehyde is the result of incomplete fermentation or the oxidation of ethanol; therefore, it can be considered to be a flaw.  On the other hand, ethyl hexanoate is an acceptable ester that is present in many ales to some extent.  For example, the Scottish and Newcastle strain throws ethyl hexanoate at above taste threshold levels (it's an integral component of the Newcastle Brown Ale flavor profile). 
Mark

Just say "no" to yeast rinsing
https://www.homebrewersassociation.org/forum/index.php?topic=19850.msg252492#msg252492

Friends don't let friends use Star San as their primary sanitizer

"Acid-anionic sanitizers are broad spectrum against bacteria and viruses, but not very effective against yeasts and molds."

Offline morticaixavier

  • I must live here
  • **********
  • Posts: 7514
  • Underhill VT
    • View Profile
    • The Best Artist in the WORLD!!!!!
Re: Bias in BJCP judging?
« Reply #16 on: March 02, 2014, 09:18:57 AM »
Seems like it would be super useful to take a tasting course with you leading. You obviously have a strong grasp of yeast metabolic byproducts and their sensory identification and that takes a lot if study, practice, and understanding. I for one would take that course
"Creativity is the residue of wasted time"
-A Einstein

"errors are [...] the portals of discovery"
- J Joyce

Offline MDixon

  • Senior Brewmaster
  • ******
  • Posts: 1275
    • View Profile
    • Mike's Homebrewing Page
Re: Bias in BJCP judging?
« Reply #17 on: March 02, 2014, 10:00:49 AM »
Again, you are flagging your experience with two people at a table as what every BJCP judge does.

Using scientific descriptors on a score sheet is poor form IMO. If the entrant doesn't relate to what you write it just isn't useful information. Writing down "Concord Grape Ester" is far more useful in the long run "ethyl heptanoate."

I'd suggest you take the online entrance exam and then the tasting exam. Also spend more time at various competitions so you have more than a singular experience.
It's not a popularity contest, it's beer!

Offline macbrews

  • Cellarman
  • **
  • Posts: 91
    • View Profile
Re: Bias in BJCP judging?
« Reply #18 on: March 02, 2014, 11:05:11 AM »
To me, at this time you are a relatively new and anonymous poster, so, why don't you give your background?  You seem awful sure of yourself.  How do you know you were right and they were wrong?  Are we witnessing the coming of the next Gordon Strong?

Putting the snarkiness aside, judging can be subjective.  BJCP judges across the board do a great job of trying to get it right and they usually do.  A head (lead) judge's job is not to disregard or change the other's perception of the beer but to come to a consensus with the other, often less experienced judge.  There are few judges out there that can get all of the nuances to every style.  I am a national BJCP judge and I have been paired with judges that have no credentials that were amazingly good and on rare occasions some that thought they knew it all, but didn't.  Most judges are aware of the shortcomings of their palates and work very hard to get it right.  In spite of inherent limitations of their senses they usually get the best 3 to the medals.

Remember, homebrewing and judging is a hobby.  There is no money awarded.  99% of the time it is for bragging rights and feedback.  When I get back 2 different interpretations of a beer that I entered, I read them both and try to see what the difference was and compare them to my notes.  I usually learn something.

Other that getting a mass spectrometer for your homebrew club, I think the system works well within its obvious limitations.

Mac

Offline ajk

  • Assistant Brewer
  • ***
  • Posts: 239
  • Foam Blowers of Indiana (FBI)
    • View Profile
Re: Bias in BJCP judging?
« Reply #19 on: March 02, 2014, 02:05:15 PM »
A well-trained judge can tell the difference between apple-like ester and acetaldehyde.  But only a minority know the former as ethyl hexanoate, and even fewer will refer to it as such.

Offline denny

  • Administrator
  • Retired with too much time on my hands
  • *****
  • Posts: 14098
  • Noti OR [1991.4, 287.6deg] AR
    • View Profile
    • Dennybrew
Re: Bias in BJCP judging?
« Reply #20 on: March 02, 2014, 02:08:17 PM »
Again, you are flagging your experience with two people at a table as what every BJCP judge does.

Using scientific descriptors on a score sheet is poor form IMO. If the entrant doesn't relate to what you write it just isn't useful information. Writing down "Concord Grape Ester" is far more useful in the long run "ethyl heptanoate."

I'd suggest you take the online entrance exam and then the tasting exam. Also spend more time at various competitions so you have more than a singular experience.

+1.060 x2 to Dixon!

Life begins at 60.....1.060, that is!

www.dennybrew.com

The best, sharpest, funniest, weirdest and most knowledgable minds in home brewing contribute on the AHA forum. - Alewyfe

Online klickitat jim

  • Official Poobah of No Life.
  • *
  • Posts: 5247
    • View Profile
Re: Bias in BJCP judging?
« Reply #21 on: March 02, 2014, 03:19:19 PM »
Typically I learn less if the teacher's goal is to show me how smart he or she is. Sometimes I just tune them out all together.

Offline S. cerevisiae

  • Senior Brewmaster
  • ******
  • Posts: 1696
  • deus ex machina
    • View Profile
Re: Bias in BJCP judging?
« Reply #22 on: March 02, 2014, 04:03:40 PM »
Using scientific descriptors on a score sheet is poor form IMO. If the entrant doesn't relate to what you write it just isn't useful information. Writing down "Concord Grape Ester" is far more useful in the long run "ethyl heptanoate."

In this instance, the problem was not that the judges could not name the chemical compound.  It's that they were too quick to judge the fruity aroma as acetaldehyde, which would have resulted in a lower score.

« Last Edit: March 03, 2014, 06:31:08 AM by S. cerevisiae »
Mark

Just say "no" to yeast rinsing
https://www.homebrewersassociation.org/forum/index.php?topic=19850.msg252492#msg252492

Friends don't let friends use Star San as their primary sanitizer

"Acid-anionic sanitizers are broad spectrum against bacteria and viruses, but not very effective against yeasts and molds."

Offline MDixon

  • Senior Brewmaster
  • ******
  • Posts: 1275
    • View Profile
    • Mike's Homebrewing Page
Re: Bias in BJCP judging?
« Reply #23 on: March 02, 2014, 05:27:09 PM »
With most judges a fruity ester would never be confused as acetaldehyde however people have different aroma and taste experiences and often perceive things at different levels. So you may pick up a phenolic and believe it is at a very high level while the judge across from you does not pick up anything. The sign of an experienced judge is one who knows at what level they pick up a particular trait and if they are hypersensitive know how to relate that to the average judge. That comes with time and experience.

It's not a popularity contest, it's beer!

Online hopfenundmalz

  • Official Poobah of No Life.
  • *
  • Posts: 6399
  • Milford, MI
    • View Profile
Re: Bias in BJCP judging?
« Reply #24 on: March 02, 2014, 08:55:49 PM »
With most judges a fruity ester would never be confused as acetaldehyde however people have different aroma and taste experiences and often perceive things at different levels. So you may pick up a phenolic and believe it is at a very high level while the judge across from you does not pick up anything. The sign of an experienced judge is one who knows at what level they pick up a particular trait and if they are hypersensitive know how to relate that to the average judge. That comes with time and experience.
Then there are those that say Bud has acetaldehyde. There are a lot of those.
Jeff Rankert
Ann Arbor Brewers Guild, AHA Member, BJCP Certified
Home-brewing, not just a hobby, it is a lifestyle!

Offline Jimmy K

  • Official Poobah of No Life.
  • *
  • Posts: 3630
  • Delaware
    • View Profile
Re: Bias in BJCP judging?
« Reply #25 on: March 02, 2014, 10:35:25 PM »
This seems to be spiraling to nowhere. Judges and judging are not perfect. It never will be, just look at controversies in judged competitions with far more concrete, measurable metrics.

Sent from my XT1030 using Tapatalk

Delmarva United Homebrewers - President by inverse coup - former president ousted himself.
AHA Member since 2006
BJCP Certified: B0958

Online klickitat jim

  • Official Poobah of No Life.
  • *
  • Posts: 5247
    • View Profile
Re: Bias in BJCP judging?
« Reply #26 on: March 02, 2014, 11:32:02 PM »
Amen. I'm on your side with this. Brand new baby judge who is just glad to be a part of it. I can only fix me, so I plan to pay attention and learn and improve along the way. Meanwhile not getting to bent about broad stroke criticism of the whole package

Offline darwin18

  • 1st Kit
  • *
  • Posts: 20
    • View Profile
Re: Bias in BJCP judging?
« Reply #27 on: March 03, 2014, 07:06:52 AM »
Did you communicate your concerns of possible biases to the competition coordinator, head judge, or did you just come onto an internet forum and post your complaints?  If you saw definite bias then it really is your responsibility to communicate to those who are in charge of and run the competition so that they can address it and ensure that it doesn't happen in future competitions.

FWIW, I would be much more concerned with the judge that had a bias against keg conditioned beers than a lack of flavor/aroma knowledge.  The first is a conscious effort while the second is just a lack of knowledge that it easily addressed.  Next time just call the head judge over and ask him his thoughts and see if he agrees.  Not every judge is an expert in every category.  In my opinion, you missed a really good opportunity to help share your knowledge of the styles with the people you were judging with. 

It's not rare either for the entries for one subcategory to score poorly against the entries in another when they're grouped into one flight.  Honestly, sometimes you just a category or subcategory where the entries are just not good.  Have you ever seen a competition results page where a 2nd and 3rd place for a flight but not a 1st?  That's an example of poor quality in that particular flight. 
BJCP #F0684 - Certified

Offline johnf

  • Brewer
  • ****
  • Posts: 434
    • View Profile
Re: Bias in BJCP judging?
« Reply #28 on: March 03, 2014, 12:36:02 PM »
From what I witnessed today, I can honestly say that it is difficult for BCJP judges remain unbiased through an entire flight.  The lead judge has a huge amount of control on the final score of a beer, and his/her biases do impact a beer's final score.  For example, a lead judge with whom I worked in one flight definitely had a bias against beers that were bottled from a soda keg.   I witnessed another table score every beer in 8C lower than every beer in 9C (both styles were grouped into the same flight).  That situation is statistically impossible without bias.

How many 8Cs were there and how many 9Cs were there. Probably not many if the categories were combined. I think you are taking some pretty egregious liberties with the word "impossible".

I do agree there are a lot false positives on acetaldehyde. I do not think this is at all limited to the BJCP. Budweiser is the perfect example of this. It does not have above threshold acetaldehyde and yet most people who know the word acetaldehyde think it does because the word is overassociated with apple aroma.

The observation about relative scoring of 8C and 9C beers strikes me as typical lack of intuition about probabilities combined with hyperbole and best and data mining at worse. If you go into a situation with the intention of observing something that validates your beliefs, you probably will.

Online hopfenundmalz

  • Official Poobah of No Life.
  • *
  • Posts: 6399
  • Milford, MI
    • View Profile
Re: Bias in BJCP judging?
« Reply #29 on: March 03, 2014, 01:00:45 PM »
From what I witnessed today, I can honestly say that it is difficult for BCJP judges remain unbiased through an entire flight.  The lead judge has a huge amount of control on the final score of a beer, and his/her biases do impact a beer's final score.  For example, a lead judge with whom I worked in one flight definitely had a bias against beers that were bottled from a soda keg.   I witnessed another table score every beer in 8C lower than every beer in 9C (both styles were grouped into the same flight).  That situation is statistically impossible without bias.

How many 8Cs were there and how many 9Cs were there. Probably not many if the categories were combined. I think you are taking some pretty egregious liberties with the word "impossible".

I do agree there are a lot false positives on acetaldehyde. I do not think this is at all limited to the BJCP. Budweiser is the perfect example of this. It does not have above threshold acetaldehyde and yet most people who know the word acetaldehyde think it does because the word is overassociated with apple aroma.

The observation about relative scoring of 8C and 9C beers strikes me as typical lack of intuition about probabilities combined with hyperbole and best and data mining at worse. If you go into a situation with the intention of observing something that validates your beliefs, you probably will.

It is often stated as fact that Bud has acetaldehyde in its aroma. As you state, it has levels below threshold.
Mitch Steele and others have stated that the Bud yeast has high levels of apple esters, which causes confusion. Go to the paragraph with the Acetaldehyde heading.
http://hoptripper.com/what-is-quality/
Jeff Rankert
Ann Arbor Brewers Guild, AHA Member, BJCP Certified
Home-brewing, not just a hobby, it is a lifestyle!