Author Topic: A question about the BJCP Judging philosophy?  (Read 6454 times)

Offline denny

  • Administrator
  • Retired with too much time on my hands
  • *****
  • Posts: 13968
  • Noti OR [1991.4, 287.6deg] AR
    • View Profile
    • Dennybrew
Re: A question about the BJCP Judging philosophy?
« Reply #60 on: July 14, 2010, 09:03:23 AM »
judging AHA sanctioned home brew events

This was mentioned before...the AHA doesn't sanction events and has nothing to do with homebrew comps beyond running NHC.
Life begins at 60.....1.060, that is!

www.dennybrew.com

The best, sharpest, funniest, weirdest and most knowledgable minds in home brewing contribute on the AHA forum. - Alewyfe

Offline pyrite

  • Brewer
  • ****
  • Posts: 443
  • Davis, CA
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #61 on: July 14, 2010, 09:17:37 AM »
judging AHA sanctioned home brew events

This was mentioned before...the AHA doesn't sanction events and has nothing to do with homebrew comps beyond running NHC.

Denny, thanks for mentioning that, although, I guess that part is somewhat confusing as well, because the literature on the AHA website specifically refers to AHA sanctioned homebrew competitions.

 "AHA/BJCP Sanctioned Competitions
Check the AHA/BJCP Sanctioned Competition Calendar to find competitions in which to enter and/or volunteer
.".

I found that at the bottom of this page..
http://www.homebrewersassociation.org/pages/competitions

If you don't get in over your head, how are you ever going to know how tall you are.

Offline bonjour

  • Administrator
  • Senior Brewmaster
  • *****
  • Posts: 1791
  • Troy, MI, 37mi, 60.9deg AR
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #62 on: July 14, 2010, 09:18:58 AM »
NHC and the COC Club Only Competition

Fred Bonjour
Co-Chair Mashing in Michigan 2014 AHA Conference in Grand Rapids, Michigan
AHA Governing Committee; AHA Conference, Club Support & Web Subcommittees



Everything under 1.100 is a 'session' beer ;)

Offline MDixon

  • Senior Brewmaster
  • ******
  • Posts: 1260
    • View Profile
    • Mike's Homebrewing Page
Re: A question about the BJCP Judging philosophy?
« Reply #63 on: July 14, 2010, 09:31:34 AM »
It is a partnership between the BJCP and the AHA. Basically it is a BJCP run program nowadays and the AHA portion is mostly publication of events, awards and the AHA provides items for the comps (books, free NHC entry, etc.) It is an excellent partnership and works very smoothly, I'd dare say almost flawlessly.
It's not a popularity contest, it's beer!

Offline beerrat

  • Assistant Brewer
  • ***
  • Posts: 185
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #64 on: July 14, 2010, 02:24:13 PM »
I don't understand why a judge would change the original score to make it fit within the other judges’ scores, within 7 points? ???

Greetings,

I like the idea of relative consensus building in home-brew competitions.  My two perspectives:

As a non beer judge - I have judged a science competition for 25 years.  Judges are volunteers.  Organization asks judges to discuss and get some agreement on the scores.  Each year I judge with different judges with different experiences.  Sometimes I'm the most experienced (either in subject matter and/or judging) and other times I am not.   Building to a rough consensus at the end ensures the participates are not impacted by me missing something in their presentation/project, being overly harsh or lenient, forgetting their grade level, or just having an off day.  And in turn, I become the double check on my fellow judges.  The outcome of this is 90% of the time is that all 3 judges are confident that the score provided and comments are representative of the work presented to them.  There is the danger of whom can filibuster their opinion, and I know there are times were we agreed to disagree.  This has been rare as most worked in the spirit of judging the participant, and not their ability to trump or be trumped by other judges.
 
As a contest competitor, I value the feedback and not sure what I would learn from getting inconsistent scores and/or comments from the judges.  

Just my perspective.
Regards,
Dan

Offline MDixon

  • Senior Brewmaster
  • ******
  • Posts: 1260
    • View Profile
    • Mike's Homebrewing Page
Re: A question about the BJCP Judging philosophy?
« Reply #65 on: July 14, 2010, 03:17:26 PM »
FWIW - Dave Housman responded to Marco and here is what he wrote (if some of it doesn't sound familiar you haven't been paying attention - read that as Extra Snarky):
Quote
There is still quite a bit of subjectivity in how one scores a beer. There are also judges with differing experience and skill levels. So when two of us, perhaps one much more experienced than another, arrive at a score for a beer, it might vary quite a bit. Perhaps one didn't recognize a fault that the other did. Or perhaps one just felt the beer was worse, or better, than than the other. To provide widely differing scores to the entrant is not nearly as informative as when the judges agree and their scores are close to each other. Any judge should be willing to work with their partner and raise or lower their scores somewhat so that they are close to one another and are sending a similar message to the entrant. Things just aren't so black and white that there isn't room for adjusting scores. Frankly as an organizer and judge I try to have judges be within 3 points whenever possible; I think 7 is still to wide a variance. So think of scoring, not just for winning something but in the feedback that score conveys. Is this a Poor, Fair, Good, Very Good or Excellent beer. Scoring is sending that message. Agreeing with your partner on where the beer falls is another way of telling the entrant what you thought of the beer. Widely divergent scores are confusing. Converging scores re-enforce the opinion.
It's not a popularity contest, it's beer!

Offline thomasbarnes

  • Brewer
  • ****
  • Posts: 413
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #66 on: July 15, 2010, 10:14:02 PM »
I don't understand why a judge would change the original score to make it fit within the other judges’ scores, within 7 points?

Call it "group-think" or consensus, as you like, but the idea is for the judges to all be within the same or adjacent scoring "bands."  If you look at a score sheet, you'll see a Scoring Guide in the lower left-hand corner. Each band is roughly a 7 point spread.

If you're consistently giving beers scores in the low teens or high 40s and the other judges are giving the same beer scores in the 20s to 30s, then there's something wrong with your calibration. You need to discuss the beer with the other judges to determine if there's something that you missed - or that they missed.


Offline thomasbarnes

  • Brewer
  • ****
  • Posts: 413
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #67 on: July 15, 2010, 10:19:33 PM »
Does anyone else think one of the Judge caricatures looks a lot like Robin Williams?

The drawings are just really bad. They look like pointillist mug shots.

Offline thomasbarnes

  • Brewer
  • ****
  • Posts: 413
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #68 on: July 15, 2010, 10:27:27 PM »
Add judges, throw out the top and bottom scores, average the rest - that's your score. Olympic style.

The problem with this idea is that there usually aren't enough judges available. Competition and judge directors have to scramble to find enough people willing to judge. This means that you're lucky if you get two judges per flight. Throw out the top and bottom scores and there are no scores left.

Offline thomasbarnes

  • Brewer
  • ****
  • Posts: 413
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #69 on: July 15, 2010, 11:22:56 PM »
The "glass ceiling" or upper limit of 43-45 with BJCP scoring needs to be re-evaluated by the BJCP Board and consideration given towards issuing a position statement to all BJCP judges regarding the upper limit for scores in the BJCP guidelines.

I believe that a perfect score of 50 is difficult to achieve for the following reasons: 1) The score sheet is broken into six sections, so there are six places where you can lose points, rather than just one. 2) The judges have no way of knowing roughly how good the next beer in the flight will be, so they naturally hold back a bit to give the next beer in the flight a chance to be slightly better. 3) The need for consensus judging makes it pretty gutsy for one judge to max out his/her score and stick to it.

Practically, most homebrew (and most commercial beer) isn't "world class" and even "world class" beer might be conceivably be improved. The example you gave suggests that a "perfect 50" for homebrew is comparable to a "perfect 10" in figure skating or gymnastic. Consider, however, that a 10.0 is extremely unusual, even in Olympic-level competition. It's not going to happen in, say, collegiate or high school level competitions. Practically, most homebrewers are at the equivalent of high-school or college athletes in terms of skill.

I HAVE judged world-class beer (several beers made by a homebrew club members which went on to medal in the NHC 2nd round, a professionally made beer which went on to win 1st place in the Brewing News National IPA Challenge). I've also run training sessions where we judge commercial examples of great beers as if they were entries in competition. I'm not ashamed to say that I gave them well-deserved scores in the high 30s to low 40s. They were great beers, but there's always some way that they could be better.

Informally, I've described the upper scoring ranges for beer as follows:

34-36 Tasty, but trivial flaws. Equivalent to most craft-brews.
37-40 Superior. No obvious flaws. Better than most craft-brews.
41-45 Outstanding. World class. Angels sing when you drink this beer.
46-49 World Champion. National Best of Show winner. Angels sing and a beam of heavenly light shines down when you drink this beer.
50      Unique. When you open this beer, the heavenly choir sings, the skies are illuminated in holy light, the finger of God points down at the beer, and a booming celestial voice proclaims, "That one."

Offline pyrite

  • Brewer
  • ****
  • Posts: 443
  • Davis, CA
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #70 on: July 16, 2010, 01:10:48 AM »
When more than one person emphasizes the same idea, well, it makes it hard not to believe.  It’s the same message that has been echoed since the start of this post “if the scores are too far off then the entrant is less likely to pay attention to what the judges are actually saying in the comments” (dbeechum).  Thereafter, a majority of you followed through with similar responses, suggesting that this BJCP judging philosophy is embedded and well understood by the people who established it, and the people who now practice it.  Ultimately, it’s all done for the better good of the people receiving the score card/comment card.

Thank you for making it all crystal clear.. 
If you don't get in over your head, how are you ever going to know how tall you are.

Offline dmtaylor

  • Senior Brewmaster
  • ******
  • Posts: 1452
  • Two Rivers, WI
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #71 on: July 16, 2010, 04:25:01 AM »
Informally, I've described the upper scoring ranges for beer as follows:

34-36 Tasty, but trivial flaws. Equivalent to most craft-brews.
37-40 Superior. No obvious flaws. Better than most craft-brews.
41-45 Outstanding. World class. Angels sing when you drink this beer.
46-49 World Champion. National Best of Show winner. Angels sing and a beam of heavenly light shines down when you drink this beer.
50      Unique. When you open this beer, the heavenly choir sings, the skies are illuminated in holy light, the finger of God points down at the beer, and a booming celestial voice proclaims, "That one."

This is so very true, I think.  Thanks for the breakdown.  It might be really helpful to folks entering their first couple of competitions.
Dave

"This is grain, which any fool can eat, but for which the Lord intended a more divine means of consumption. Let us give praise to our Maker, and glory to His bounty, by learning about... BEER!" - Friar Tuck (Robin Hood - Prince of Thieves)

Offline MDixon

  • Senior Brewmaster
  • ******
  • Posts: 1260
    • View Profile
    • Mike's Homebrewing Page
Re: A question about the BJCP Judging philosophy?
« Reply #72 on: July 16, 2010, 05:29:28 AM »
I am not in complete agreement. As a person who samples a vast amount of beer (home and craft), I think Thomas has a really high valuation of craft beer. While much of it is really good, there is a vast majority which could stand improvement. I also don't give out 37's to beers just because they have "no noticeable flaws". A beer can be flawless (in terms of off flavors and aromas) and still miss style. Take a beer we have all had and probably enjoy Sierra Nevada Pale Ale. Probably the most recognized and maybe the benchmark of the APA style. If it is fresh it would be a solid 36 in my world and we all would agree it would generally be without off flavors and aromas. It may just be semantics to some, but I tend to think of stylistic inaccuracies as a different animal while I tend to think of flaws = faults http://www.bjcp.org/faults.php

My breaking points are somewhat different, for instance angels don't sing till a 43 is reached  ;)
It's not a popularity contest, it's beer!

Offline mrcceo

  • Cellarman
  • **
  • Posts: 56
  • Location - Philadelphia Area
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #73 on: July 16, 2010, 07:48:51 AM »
I have been following this thread along with the responses on the AHA site and wanted to offer the following:  If your going to maintain the practice of the 7 point spread the BJCP should think about only giving out one score sheet which renders a decision based on the majority consensus of the judges and written in a clear concise manner by the highest ranking judge.  The problem for me in interpreting the results of the score sheets is that often the comments don’t relate to, or support the score.  I now understand why. It’s very confusing to read a judge’s comments which praise your entry and see a score which is not in line with their remarks.  To my way of thinking I would rather read one score sheet which makes sense and does not leave me feeling as though I got to the end of a book and the author gave me 3-4 different endings and told me to pick one.

That being said there is also the issue of the credentials of the judges. I have received score sheets which were all prepared by novice judges or had no judge rating at all on the sheet.

The system doesn’t seem broken but when you’re dealing with subjective opinions and the entrant is looking for what I believe is sagely feedback and a fair balance critique they shouldn’t be receiving uncertainty.

Just my thoughts.

Offline pyrite

  • Brewer
  • ****
  • Posts: 443
  • Davis, CA
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #74 on: July 16, 2010, 09:45:56 AM »
I have been following this thread along with the responses on the AHA site and wanted to offer the following:  If your going to maintain the practice of the 7 point spread the BJCP should think about only giving out one score sheet which renders a decision based on the majority consensus of the judges and written in a clear concise manner by the highest ranking judge.  The problem for me in interpreting the results of the score sheets is that often the comments don’t relate to, or support the score.  I now understand why. It’s very confusing to read a judge’s comments which praise your entry and see a score which is not in line with their remarks.  To my way of thinking I would rather read one score sheet which makes sense and does not leave me feeling as though I got to the end of a book and the author gave me 3-4 different endings and told me to pick one.

That being said there is also the issue of the credentials of the judges. I have received score sheets which were all prepared by novice judges or had no judge rating at all on the sheet.

The system doesn’t seem broken but when you’re dealing with subjective opinions and the entrant is looking for what I believe is sagely feedback and a fair balance critique they shouldn’t be receiving uncertainty.

Just my thoughts.


So let it be said, so let it be heard.
If you don't get in over your head, how are you ever going to know how tall you are.