Author Topic: A question about the BJCP Judging philosophy?  (Read 6620 times)

Offline MDixon

  • Senior Brewmaster
  • ******
  • Posts: 1276
    • View Profile
    • Mike's Homebrewing Page
Re: A question about the BJCP Judging philosophy?
« Reply #15 on: July 09, 2010, 02:39:45 PM »
Well, I might know how you can get in touch with the Communications Director  ::)

You can reach any of the BJCP officers on this page
http://www.bjcp.org/officers.php
« Last Edit: July 09, 2010, 02:42:51 PM by MDixon »
It's not a popularity contest, it's beer!

Offline pyrite

  • Brewer
  • ****
  • Posts: 443
  • Davis, CA
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #16 on: July 09, 2010, 02:46:55 PM »
thank you.
If you don't get in over your head, how are you ever going to know how tall you are.

Offline EHall

  • Brewmaster
  • *****
  • Posts: 608
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #17 on: July 09, 2010, 04:41:40 PM »
Somehow I knew you were going to ask that!  ;) For the life of my I can't find the link, doesn't even look like they advertise it on the homebrewing site anymore... I'm sure someone else will have it handy, but I keep my email pretty clean and I don't have any
emails from them right now, if no one comes up with it, I'll respond on Monday.
Phoenix, AZ

Offline tom

  • Senior Brewmaster
  • ******
  • Posts: 1110
  • Denver, CO
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #18 on: July 09, 2010, 05:19:13 PM »
Not everyone has a perfect palate. I am very sensitive to diacetyl and fusel alcohols. Not so sensitive to oxidation or acetaldehyde. So if another judge tastes those, I can adjust my scoresheet accordingly.
Brew on

Offline dmtaylor

  • Senior Brewmaster
  • ******
  • Posts: 1491
  • Two Rivers, WI
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #19 on: July 09, 2010, 05:34:06 PM »
I am a Recognized BJCP judge, and I for one ABHOR the idea of being virtually forced to change a score based on an arbitrary requirement to be within 7 or 5 or 3 points of another judge, who may have a lot more or a lot less experience -- doesn't matter.  Everyone should be entitled to their own opinion, unadulterated, then let the receiver of the scoresheets decide for himself/herself which judge is right on and which he/she does not agree with.  Better yet -- get a third and even fourth opinion if there is a broad difference of opinion that cannot be resolved.

I feel it is most wise to score each beer on your own without discussion, then spend a few minutes comparing notes to see if there is something you missed or to see if maybe you scored a little too low or too high.  That much is reasonable.  But also, don't let anyone convince you that you scored incorrectly and that "surely you can taste that diacetyl" or DMS or whatever, even if you know in your heart that you can't pick it up in the slightest.  Be honest!  Don't let anyone tell you that YOUR score is wrong!  You own it!  You sign your name to it -- no one else does!  Any changes must be because YOU realized that you want to change your score, NOT because of some arbitrary range rule.  To be coerced into changing a score because others tell you that their opinion matters more than your own constitutes LYING in my book if you do change that score.

I believe this very firmly.  I am a man of integrity.  And like most judges, I am well educated on beer styles and have tasted more than my share.  This is a case where sometimes judges just need to agree to disagree.
Dave

"This is grain, which any fool can eat, but for which the Lord intended a more divine means of consumption. Let us give praise to our Maker, and glory to His bounty, by learning about... BEER!" - Friar Tuck (Robin Hood - Prince of Thieves)

Offline pyrite

  • Brewer
  • ****
  • Posts: 443
  • Davis, CA
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #20 on: July 09, 2010, 06:10:39 PM »
Not everyone has a perfect palate. I am very sensitive to diacetyl and fusel alcohols. Not so sensitive to oxidation or acetaldehyde. So if another judge tastes those, I can adjust my scoresheet accordingly.

I don't think anyone person has a perfect palate. That is why I whole heartedly believe keeping the independent voice/comment card and withholding changes to the comment card/score card when consulting with other judges is vital to the integrity of the AHA homebrew competition.   When you go to the beer specialty store and are looking at the selection of beer in the fridge only you know what you like (of course your not consulting the BJCP guidelines). I am stating that if you did not notice something in that beer, then it is either not that noticeable or you pallet is not that tuned in to it, and that’s what I think to be a valuable attribute of a competition and to each person who pays to have their beer objectively and subjectively evaluated (an independent evaluation according to the BJCP guidelines without the influence of some other judge).   
If you don't get in over your head, how are you ever going to know how tall you are.

Offline pyrite

  • Brewer
  • ****
  • Posts: 443
  • Davis, CA
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #21 on: July 09, 2010, 06:18:19 PM »
I also occasionally encounter those who just won't give a score higher than 41 or 42, even if it is really good, because they perceive that a score higher than that just doesn't get awarded, and, if it does, then it pretty much guarantees that the beer they've awarded a score of 41+ is going to win the comp.  This is more of an issue during intra-club comps with just one round, and some judges may have entered a beer as well--in other words, they have "skin in the game."  Even though they aren't judging their own entry, they're still judging in that style--something that wouldn't happen in a real BJCP competition.

I sympathize with you; it is frustrating to have beers judged by those with little experience and/or no BJCP certification saying giving a lower than expected score and very little in the way of constructive feedback, and then getting a vastly different score at a different competition (along with a medal).  I'd suggest having your beers judged outside the homebrew club at one or two competitions by those who have no "skin in the game" before you make any conclusions about your beer.  JMO.

I've never heard of or thought of that point of view, but it is an interesting one.
If you don't get in over your head, how are you ever going to know how tall you are.

Offline pyrite

  • Brewer
  • ****
  • Posts: 443
  • Davis, CA
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #22 on: July 09, 2010, 06:31:15 PM »
I am a Recognized BJCP judge, and I for one ABHOR the idea of being virtually forced to change a score based on an arbitrary requirement to be within 7 or 5 or 3 points of another judge, who may have a lot more or a lot less experience -- doesn't matter.  Everyone should be entitled to their own opinion, unadulterated, then let the receiver of the scoresheets decide for himself/herself which judge is right on and which he/she does not agree with.  Better yet -- get a third and even fourth opinion if there is a broad difference of opinion that cannot be resolved.

I feel it is most wise to score each beer on your own without discussion, then spend a few minutes comparing notes to see if there is something you missed or to see if maybe you scored a little too low or too high.  That much is reasonable.  But also, don't let anyone convince you that you scored incorrectly and that "surely you can taste that diacetyl" or DMS or whatever, even if you know in your heart that you can't pick it up in the slightest.  Be honest!  Don't let anyone tell you that YOUR score is wrong!  You own it!  You sign your name to it -- no one else does!  Any changes must be because YOU realized that you want to change your score, NOT because of some arbitrary range rule.  To be coerced into changing a score because others tell you that their opinion matters more than your own constitutes LYING in my book if you do change that score.

I believe this very firmly.  I am a man of integrity.  And like most judges, I am well educated on beer styles and have tasted more than my share.  This is a case where sometimes judges just need to agree to disagree.

For the most part I agree with what you have to say.  In the BJCP judge study guide it states, that even if a master judge has an opinion that is different than your, you don't have to agree with their opinion.  However, it states that you have to come to a consensus score within 7 points of each other.  I got to say though; I can't wait to get some homebrew comp judging experience to understand why that is.
If you don't get in over your head, how are you ever going to know how tall you are.

Offline richardt

  • Senior Brewmaster
  • ******
  • Posts: 1227
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #23 on: July 09, 2010, 06:54:29 PM »
Another thing all judges need to be aware of is their own bias (or biases).  There are other judges/brewers whom I greatly respect that just tend to favor say, Big Belgians and Wild Ales over Milds and Pale Ales or hoppy beers over malty beers or whatever.  If you're really into entering your beer into competitions--you need to take a page from the Speech class you took in school:  "consider [the biases of] your audience."  Listen to what others say (from what you read online or in books and from what you hear at club meetings or beer comps) about key features of each beer style as well as what styles tend to win in competitions.  Review the various winners of each category as well as overall winners of each competition to look for common themes.  Review their recipes to look for common as well as unusual ingredients.  There is a tendency for winners to be excellent examples of big beers or less common styles.  That's why I think BOS rounds are total crapshoots sometimes.  You never know if your BOS judges are proponents of Session ales over Saisons or Lagers over Lambics.  But a well-made beer that fits the style should always do well in it's category.

Offline hokerer

  • I spend way too much time on the AHA forum
  • ********
  • Posts: 2639
  • Manassas, VA
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #24 on: July 09, 2010, 07:27:28 PM »
Does anyone else think one of the Judge caricatures looks a lot like Robin Williams?

Raises hand.  Every time I get to that part of the mag, Robin Williams is exactly who I think of.
Joe

Offline texlaw

  • 1st Kit
  • *
  • Posts: 8
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #25 on: July 09, 2010, 07:38:04 PM »
The 7-point rule really is not directly about credibility or dishonesty or manipulation or pride, or anything like that.  It is, in large part, about judging the beer properly and giving the entrant a fair score.

As has been mentioned, a great deal of subjectivity goes into judging a beer, but as also has been mentioned, a great deal of effort has gone into limiting that subjectivity.  If two judges cannot score a beer within 7 points, then too much subjectivity must be present.  That is when the judges need to discuss what is going on and the reasons for each score.  Sometimes, it is a matter of "I like this one more," sometimes it is a matter of "I can't pick that up," sometimes it is a matter of interpreting the guidelines, and sometimes it is something completely unexpected.  Usually, a brief discussion resolves the issue.

Experience plays a large part, as well.  Each judge has to know his own strengths and weaknesses.  I know that I am hypersensitive to DMS, so I have to be careful about how hard I ding someone when I pick up DMS inappropriately.  I also know that I am hyposensitive to the flavor and aroma of diacetyl.  While I usually can pick up the diacetyl mouthfeel, I need to listen up when my fellow judge mentions diacetyl, while I do not detect it.  Experience teaches you how to deal with your own palate.  Those judge discussions also, often, are some of the most valuable in one's education as a beer judge.

In the end, the judges make up a panel, not just a survey of individuals.  If the panel cannot arrive to a consensus of 7 points (a span of 14% of the entire scoring range, for crying out loud), then something is wrong with the panel, and the Judge Coordinator needs to get involved.  I have seen that happen at least a few times, most notably when two judges could not agree on whether the beer had an acetobacter infection (one judge believed so, and the other thought it was "yeast character").  They took it to the Judge Coordinator, a couple other experienced judges were brought over, and the matter was resolved.  That is just a matter of getting a fair score and a fair evaluation to the entrant.  I hate to think what the entrant would have thought had he received two scoresheets with very different scores and notes.  I also hate to think what would have happened had the incorrect judge continued to judge beers incorrectly in the same manner.


Doak Procter
BJCP National

Offline MDixon

  • Senior Brewmaster
  • ******
  • Posts: 1276
    • View Profile
    • Mike's Homebrewing Page
Re: A question about the BJCP Judging philosophy?
« Reply #26 on: July 10, 2010, 04:49:37 AM »
Dave - while your comments are YOUR comments, you do the entrant a disservice to not work to a consensus score. The goal is to evaluate every beer in the flight the same way and for the flight to be able to stand alone. If you (or your fellow judge) is not working to achieve a consensus score within 7 points (3 in my world) then you aren't giving all the beers in the flight adequate attention. Like it or not, you and I if on a flight together would sit at the table until we reached a consensus score which was acceptable.

Everyone please review the Judge Procedures Manual which was part of the review material for the BJCP exam if you have taken the exam in the past 4-5 years. http://www.bjcp.org/judgeprocman.php
Quote
Once all of the judges at the table have finished filling out their score sheets, they should discuss the entry and their scoring. Keep in mind that some people are more sensitive to certain flavor and aroma compounds (such as diacetyl or DMS) than others. Scores should be within seven points, and preferably within five or fewer points. Adjust scoring as necessary, and write the final assigned score on the cover sheet and flight sheet. Final assigned scores are not necessarily an average of the individual scores.
It's not a popularity contest, it's beer!

Offline richardt

  • Senior Brewmaster
  • ******
  • Posts: 1227
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #27 on: July 10, 2010, 06:04:36 AM »
My earlier rants about BJCP judging experiences are just that and they represent less than 5-10% of my experiences when judging.  I have to say that >90-95% of the time, the judges are really trying their best to be fair, and are considerate of each other when discussing evaluations, reconciling scores.  They also take the time to identify, describe, and educate each other with regard to style characteristics or any perceived off-flavor(s)/aroma(s) and provide constructive feedback to the entrant.  I do think the BJCP program should focus more on sensory training and critiquing than on rote memorization of beer stats and beer literacy for the written exam.  I favor the BJCP exam evolving to a multiple choice format (less subjective, ability to assess more areas of knowledge-base, and quicker turn-around time for exam grading) and more emphasis put on sensory training and judging skills.

Offline dmtaylor

  • Senior Brewmaster
  • ******
  • Posts: 1491
  • Two Rivers, WI
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #28 on: July 10, 2010, 06:56:53 AM »
Dave - while your comments are YOUR comments, you do the entrant a disservice to not work to a consensus score. The goal is to evaluate every beer in the flight the same way and for the flight to be able to stand alone. If you (or your fellow judge) is not working to achieve a consensus score within 7 points (3 in my world) then you aren't giving all the beers in the flight adequate attention. Like it or not, you and I if on a flight together would sit at the table until we reached a consensus score which was acceptable.

Mike, of course we need to TRY to reach a consensus.  90+% of the time, this is very easy to do.  I don't know if I've ever had an instance where I could not easily and reasonably reach consensus with another judge, but I'm certain that it does happen often enough where one guy finally gives up because he loses the war of wills.  There are times when consensus is very difficult to achieve, and who's to say which judge is right or wrong?  THAT is what bugs me.

My primary concern is that this turns into a battle of whose schwartz is longer than the other's, when in fact, it is pretty close to a 50/50 chance that the guy with the shorter one is right, and the longer one is wrong.  I'll give you a specific scenario, based on what so often happens in a real competition -- First, let's say that in any other universe, if you had 100 certified judges, who were completely independent and without consensus or bias, and took the overall average score for a particular beer, the true average score for this beer is 32.  The scores of those 100 judges actually ranged from somewhere around 20 to 40, but the real score should be 32.  So, then, move back to the real world, in a real competition, and pick any one of those certified judges, and pair him up with an apprentice who might really know a lot about that particular style, or might not.  So then the apprentice judge gives this beer (which deserves a 32) a 29 -- not bad for being in close range to the "true" score -- and the other judge, who happens to be a National rank, decides that it is a 21 for whatever reason -- maybe he has a head cold, or he's drunk from two previous flights of Belgian strongs, or whatever.  My concern is that the apprentice guy will very likely feel pressured into lowering his score to a 25 or so, whether self-induced or by the National; when in reality, he was right on at a 29!  By forcing the two judges to come to a consensus to some arbitrary range, it forced the score in the WRONG direction!!  I feel this happens VERY often, FAR too often -- it's virtually a 50/50 chance!!!  How is this doing anyone any service?!?!

It is easy to insist on changing someone else's score when you are of higher rank than the other guy.  Please take this into consideration -- just because your schwartz is longer, doesn't mean you are omniscient.  I am not directing this at you personally -- I have zero reason to doubt that you are anything but an excellent judge.  But surely you will agree that there are judges out there who are very pretentious and stubborn, while there are infinite others who may have very little experience in BJCP, but that doesn't mean they are complete morons either!  In fact many of the inexperienced guys may have better palates than the Nationals.

Schwartz ain't everything.  Peace, out.
Dave

"This is grain, which any fool can eat, but for which the Lord intended a more divine means of consumption. Let us give praise to our Maker, and glory to His bounty, by learning about... BEER!" - Friar Tuck (Robin Hood - Prince of Thieves)

Offline MDixon

  • Senior Brewmaster
  • ******
  • Posts: 1276
    • View Profile
    • Mike's Homebrewing Page
Re: A question about the BJCP Judging philosophy?
« Reply #29 on: July 10, 2010, 10:43:12 AM »
If you really want to know what chaff's my butt it isn't having to come to a consensus score, that doesn't bother me in the least, it is the imposed courtesy lower limit of 13 and the upper limit which tends to max out at 45. So in reality 18 out of 50 points are never utilized. At one time when less information was available about brewing the 13 so new brewer's wouldn't give up brewing seemed appropriate. Now I am not so sure. In the competitions I've judged this year I've seen at least 3 or 4 entries which could not score more than a 9 and I had to make up some points to bring them to 13. The upper level is more or less self imposed, but seems to be the thought process with everyone. (FWIW, I have given a 48 to a metheglin).
It's not a popularity contest, it's beer!