[EHall]

Here's that email address for the email forum...

techtalk@brewersassociation.org

[pyrite]

Here's that email address for the email forum...

techtalk@brewersassociation.org

Thanks EHall, the email has been sent.

[MDixon]

Just so we are clear, TechTalk has nothing to do with the BJCP other than being another outlet for AHA member discussion. While the AHA and the BJCP work together on competition registration, the BJCP is in no way governed by the AHA. About the only thing posting on TechTalk will do is add more discussion from others in the BJCP who are also part of the AHA.

[EHall]

I suggested he submit his question to the email forum due to Gordon and David frequent that forum, he'll hopefully get a more 'complete' answer from some high level judges.

[pyrite]

That forum you guys are talking about seems like a secret society forum that only the higher-ups get access to.  I didn't know that email was to some other forum, I thought that email address belonged to this forum.  Either way, this was the email I sent out last night. Thanks again guys.

To whom it may concern,

I'm preparing to take the BJCP judge certification exam and in doing so I'm reading the BJCP Exam Study Guide, on page 14 it states "when the other judges have finished scoring the beer, discuss the technical and stylistic merits of the beer, and arrive at a consensus score.  Be prepared to adjust your score to make them fall within 7 points of the other judges at your table".?

I don't understand why a judge would change the original score to make it fit within the other judges’ scores, within 7 points?

I started this thread on the AHA website a week ago and I am still uncertain how this BJCP judging philosophy was adopted.  I would appreciate any thoughts you would like to add so that I could better understand the, benefits of modifying a score card so that they look aligned.  We at the forum have been in discussion on this topic and would love to have some insight from the people who crafted and or understand this Judging system.

http://www.homebrewersassociation.org/forum/index.php?topic=2784.msg31720#msg31720

Thank you,

pyrite

[bonjour]

I can't let Mike take all the heat here.

When the other judges have finished scoring the beer, discuss the technical and stylistic merits of the beer and arrive at a consensus score. Be prepared to adjust your scores to make them fall within 7 points of the other judges at your table.

Are you 100% knowledgeable of the style that you are judging?
Are you able to perceive 100% of all flavor and aroma characteristics in beer?  At a low threshold level?
Are you able to identify and list all flavor and aroma components in beer with 100% accuracy?
Are you 100% familiar with the variability of the style?
Have you tried ALL the classic examples at various stages of freshness and at the brewery that made them?

If the answer is no, then why wouldn't you listen to the other judges at the table.  This is where a lot of beer education occurs.
Consensus means to listen to all the input and see if you feel that you are missing something and adjust your score to compensate.  Because you are blind to diacetyl doesn't mean that an excessive amount isn't there.  Part of the consensus process is to determine why your point spread is off. 

I like to be within 5 points, not 7, a bit more than Mike's 3

I may not be a GM, but like Mike I'm listed on the BJCP Officer's page.

[pyrite]

I can't let Mike take all the heat here.

When the other judges have finished scoring the beer, discuss the technical and stylistic merits of the beer and arrive at a consensus score. Be prepared to adjust your scores to make them fall within 7 points of the other judges at your table.

Are you 100% knowledgeable of the style that you are judging?
Are you able to perceive 100% of all flavor and aroma characteristics in beer?  At a low threshold level?
Are you able to identify and list all flavor and aroma components in beer with 100% accuracy?
Are you 100% familiar with the variability of the style?
Have you tried ALL the classic examples at various stages of freshness and at the brewery that made them?

If the answer is no, then why wouldn't you listen to the other judges at the table.  This is where a lot of beer education occurs.
Consensus means to listen to all the input and see if you feel that you are missing something and adjust your score to compensate.  Because you are blind to diacetyl doesn't mean that an excessive amount isn't there.  Part of the consensus process is to determine why your point spread is off.  

I like to be within 5 points, not 7, a bit more than Mike's 3

I may not be a GM, but like Mike I'm listed on the BJCP Officer's page.

Bonjur, I am not an expert beer taster (not even close), and I do understand that there are other people that have experienced more flavors than the small amount that I have had the opportunity to taste.  I think you make a very good and clear point. However, I just want to make something clear; I’m not trying to give anyone any heat. If you feel like I am a little too stern in my posts while I search for reasons of why the BJCP governing body has adopted this certain judging philosophy, well I apologize.  However, before I can be part of and take part in judging AHA sanctioned home brew events, when I come across something I don’t understand such as the changing of score cards, I want to understand why this is practiced.  I’m just not used to being led blindly and practicing something I don’t understand.  

Some people have posted reason why the judging system that is in place is beneficial, including yourself, and I respect those opinions, and it has made the confusion a lot clearer.  Conversely, other forum members have discussed the benefits of not changing score cards.  They argue that people have different sensory perceptions, thus, tasting things differently and picking up different flavor (and this is what I find to be the beauty in having different people judging one beer that not everyone has the same evaluation). However, this is where it gets tricky, because to become a certified judge, one must pass an extensive exam. During the exam there are 4 different styles of beers one must judge correctly to pass, the exam does not allow a consensus score to be reached.  The exam tests a person on how well they can independently score, give feedback and critique a beer free of consensus scoring.

[bonjour]

Perhaps I was too strong in using the word heat,  sorry about that.

Both Mike and I conduct classes to prep brewers for the EXAM

On the exam the 4 beers are scored by at least 2 proctors, they are NOT allowed to change their scores.  A consensus score is determined after they have evaluated the beer and this is the score that you are compared to.  This consensus score is evaluated by the exam directors and if needed (not often) it is adjusted.  The purpose of the proctors is to taste teh beers and explain what they tasted so the scorers can evaluate how well your perceptions compare to what is in the beer.  All of this is evaluated to come up with your tasting score.

When I first start teaching the class there are always a few outliers because the brewer had trouble picking out flavors/aromas.  By the time the class ends the class presents a fairly tight number (usually within the 7).

[beerocd]

Add judges, throw out the top and bottom scores, average the rest - that's your score. Olympic style. Only problem is now everyone has to submit more beer. That would be consensus without any alpha-judge type activities going on. Only the comments and the average score go to the brewer, the individual scores are on a tear off, like a diner receipt and are discarded after the event.

[tapper]

On the exam the 4 beers are scored by at least 2 proctors, they are NOT allowed to change their scores.  A consensus score is determined after they have evaluated the beer and this is the score that you are compared to.  This consensus score is evaluated by the exam directors and if needed (not often) it is adjusted.  The purpose of the proctors is to taste teh beers and explain what they tasted so the scorers can evaluate how well your perceptions compare to what is in the beer.  All of this is evaluated to come up with your tasting score.

my beef with this is when i took my exam (4 months ago) the 2 proctors on our Old Ale beer gave the beer a 13 and a 30.  If the two proctors can't be closer than 17 points and they are not allowed to change there scores how are we, as examinees, being fairly judged on that beer? seems like the two proctors would be able to come closer on scores than that, i don't think the consensus score was adjusted.
i do have to say in regards to getting our scores back, ours came back fairly quickly, 3 months, so props to the BJCP graders for that.
lots of good (and helpful) discussions going on. 

[richardt]

IMO, the written portion is like the "barbed wire fence" that keeps casual craft-beer drinkers and homebrewers from getting (more) involved in the BJCP.  And the sensory portion is the achilles heel.  I believe more emphasis should be put on putting out a BJCP sensory guidelines (just like BJCP style guidelines), along with providing AHA clubs and BJCP exam applicants a sensory training kit and on-line sensory tutorial program.  IMO, the "literacy" aspect is over-emphasized.  I'm not necessarily saying make the exam "easier" from a scoring standpoint, just more objective and easier to take and grade (MC and TF).

IMO, the knowledge base and the sensory training are the most important, not the writing.  If you don't know what you need to be sensing with regards to aroma, appearance, flavor, mouthfeel, etc., and you have no prior sensory training, then the BJCP style guidelines and "literacy" isn't going to amount to much.  I get really disappointed when I get scoresheets back from judges who can't really describe what they're sensing (and I think it is because they lack the sensory experience) or they incorrectly parrot some phrases right out of the style guidelines.  The Siebel kits are a good start, but more thought and discussion ought to go into improving the sensory abilities of the individual BJCP judges on the individual, club, and AHA circuit levels.

[bonjour]

my beef with this is when i took my exam (4 months ago) the 2 proctors on our Old Ale beer gave the beer a 13 and a 30.  If the two proctors can't be closer than 17 points and they are not allowed to change there scores how are we, as examinees, being fairly judged on that beer? seems like the two proctors would be able to come closer on scores than that, i don't think the consensus score was adjusted.
i do have to say in regards to getting our scores back, ours came back fairly quickly, 3 months, so props to the BJCP graders for that.
lots of good (and helpful) discussions going on. 

That's a tough one, and I don't have all the info.  The exam admin is also required to state what the 4 beers were and anything notable about them.  Spiked, classic example (name) etc.  so that helps.  I also prefer to use 3 proctors, to allow more choices when something like this happens.  I'm sure the exam directors talked a lot about that one to make it as fair as possible. 

[MDixon]

I agree, 3 proctors is better. The reason for them not changing the scores is so the examinees get the best possible outcome. The graders will look at the individual scores and the consensus scores and may decide to grade the taste portion in a different manner to accommodate for the difference. In reality, the scoring of the beers is a wildcard and the VERY WORST one can score is 9/20. (100 total points available for the taste portion which is 30% of the overall score). http://www.bjcp.org/docs/Scrguide-2010.pdf
[FWIW - I scored a 91 on taste and lost 5 points due to scoring inaccuracy vs the proctors.]

E - if you view the links I posted he would have had their DIRECT email addresses, not simply another forum 

[MDixon]

richard - you just cannot get people to write what they sense on the page. I've sat across the table with numerous novice judges and while they don't wax poetic, they do have good comments about the beers, they just don't have the cohones to put the verbiage on the page. It doesn't do much good when one senses something if they don't write it down!

[richardt]

Mike,

Interesting point.  You are right about that behavior by some novices. 

I see it both ways.  On the one hand, a novice may be "afraid to commit" to an answer and prefers to remain vague or list a bunch of options in an attempt to "not look dumb."  On the other hand, and as Fred pointed out earlier, knowledge and experience helps a judge search for the key features and flaws of a particular beer style and assess them accordingly.  Ideally, a more experienced judge would pass along that knowledge to a novice judge during a judging session if the novice failed to detect the character or flaw or scored the exam in a vastly different way.

It brings to mind two aphorisms gleaned from my mentors during my years of formal education and training:
1.)  Commit yourself to a diagnosis/decision/plan of action.  You can't help someone until you do!  Most of the time you'll be right; but, if evidence arises later on that indicates you were wrong, or another plan of action proves to be more appropriate, be willing to change.
2.)  You only see what you look for; You only look for what you know.  There's no substitute for knowing your stuff and being experienced.  If you don't know what you don't know, you won't detect it or know what to do with it if you do detect it.

That's why I feel strongly that the sensory training should be more robust and the knowledge base be thoroughly assessed via a MC and TF format--150 questions can easily be asked and answered in 3 hours by any applicant who has studied and prepared for the exam.