Author Topic: A question about the BJCP Judging philosophy?  (Read 6311 times)

Offline richardt

  • Senior Brewmaster
  • ******
  • Posts: 1227
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #30 on: July 10, 2010, 11:16:09 AM »
The "glass ceiling" or upper limit of 43-45 with BJCP scoring needs to be re-evaluated by the BJCP Board and consideration given towards issuing a position statement to all BJCP judges regarding the upper limit for scores in the BJCP guidelines.

IMO, a score of 50 should be possible to achieve.

I know of no other competition involving subjective judging of objective performances (e.g., diving, gymnastics, snow-boarding) that does not allow a "perfect" score to be given.
A brewer's 12 ounce bottled performance deserves the same chance.

I also note how the BJCP exam itself is damn near impossible to break above 90, as well; especially given its format (written exam) and time limitations.  Over 5 thousand bright homebrewers have taken the exam and only 88 people "subjectively" scored in the 90's on the exam and met the experience criteria to become a "Master" BJCP judge.  The time has come to make the exam easier to take, easier to grade, and easier to learn how to be a good judge. 

Offline MDixon

  • Senior Brewmaster
  • ******
  • Posts: 1241
    • View Profile
    • Mike's Homebrewing Page
Re: A question about the BJCP Judging philosophy?
« Reply #31 on: July 10, 2010, 06:22:42 PM »
Any "glass ceiling" is self imposed (I posted as much). There is no BJCP mandate that 45 is the highest possible.

- -

Lot's of people blow smoke, few really want to help build the fire. If the exam is something you are passionate about changing, then offer your assistance, not your advice, YOUR ASSISTANCE to the the Exam Directorate. That assistance could be in the form of M/C questions on styles, recipes, troubleshooting, etc.

It almost sounded like you believe more people should be Master. Aw heck, it did sound that way. Is it you dislike the difficulty in achieving a score higher than 90 or you dislike the number who have made that grade and have also achieved the required number of experience points?
« Last Edit: July 10, 2010, 06:25:04 PM by MDixon »
It's not a popularity contest, it's beer!

Offline richardt

  • Senior Brewmaster
  • ******
  • Posts: 1227
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #32 on: July 10, 2010, 08:10:42 PM »
MDixon (Nationally-Ranked, BJCP Judge, BJCP Communications Director),

Glad to hear that you perceive the "glass ceiling" to be self-imposed.  I'll be juding in a comp at the end of the month.  I'll be glad to put a 49 or 50 on someone's entry if it merits it.  Thanks for the clarification.  I shall quote you in any discussions I have with my fellow judges should there be more than a 3-, 5-, or 7- point difference between judges and the argument arise that "a score that high is never given."

in re assistance, I'd be happy to follow through with actions. 
To whom shall I send my list of suggested questions for the BJCP exam?
Please supply a contact name and e-mail address for all of us on the forum (you may wish to start another post).

All I ask is that, if I spend the time in good faith creating and submitting these questions, that they are going to be seriously considered along with a format change towards an objective examination format.  Can you guarantee that?  If not, I'm a busy man and I'm not going to waste my time.  I'll be even more to the point--If you're asking me to join the BJCP exam directorate just to grade more written exams, my answer is a firm "No, thank you."

I'd like more people to be educated and competent beer judges.  Yet, truthfully, a lot of people struggle with the current format;  and many won't even attempt BJCP certification because of it.  I don't have the desire to get into an "old school vs new blood" arguement, i.e., "I had to do time limited written exams, so you do too."   What I don't like is the subjectivity.  I do appreciate the thoroughness and constructive feedback of the individual(s) who graded my BJCP exam.  The 6-month wait to receive my results---not so much.  There are plenty of ways to make it more objective and accomplish the desired goals of the BJCP certification program.  I do think experience counts--I do not have a problem with it. 

My point about the number of master judges (88) simply has to do with the fact that most Master Judges only score 90-92 on the exam.  Really?  Not one individual ever "aced" it?  Come on, ...Really?  There are people who "ace" the SAT's!  This tells me that the BJCP grading process and/or exam format is flawed.  Just like I perceive the scoring process to be at some beer competitions.  I'm not the only one--I didn't start this thread.

Richardt (Certified-Ranked BJCP judge)

Offline pyrite

  • Brewer
  • ****
  • Posts: 443
  • Davis, CA
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #33 on: July 10, 2010, 08:35:00 PM »
I think all the people who have posted on this thread have valid points, and it's very interesting to hear these different point of views.

If you don't get in over your head, how are you ever going to know how tall you are.

Offline a10t2

  • Official Poobah of No Life.
  • *
  • Posts: 3582
  • Ask me why I don't like Chico!
    • View Profile
    • SeanTerrill.com
Re: A question about the BJCP Judging philosophy?
« Reply #34 on: July 10, 2010, 09:04:33 PM »
a format change towards an objective examination format.

I love the idea, but how would that work? Eliminate the tasting portion?
Beer is like porn. You can buy it, but it's more fun to make your own.
seanterrill.com/category/brewing
twomilebrewing.com

Offline beerocd

  • Senior Brewmaster
  • ******
  • Posts: 1429
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #35 on: July 10, 2010, 09:51:39 PM »
It may be easier to start a new organization than change the current one. Just sayin' ...  :-\
The moral majority, is neither.

Offline MDixon

  • Senior Brewmaster
  • ******
  • Posts: 1241
    • View Profile
    • Mike's Homebrewing Page
Re: A question about the BJCP Judging philosophy?
« Reply #36 on: July 11, 2010, 05:06:51 AM »
Glad to hear that you perceive the "glass ceiling" to be self-imposed.  I'll be juding in a comp at the end of the month.  I'll be glad to put a 49 or 50 on someone's entry if it merits it.  Thanks for the clarification.  I shall quote you in any discussions I have with my fellow judges should there be more than a 3-, 5-, or 7- point difference between judges and the argument arise that "a score that high is never given."
It is self imposed. I'm received a 48 at comp before. It wasn't warranted, but it has happened.
Quote from: richardt
in re assistance, I'd be happy to follow through with actions. 
To whom shall I send my list of suggested questions for the BJCP exam?
Please supply a contact name and e-mail address for all of us on the forum (you may wish to start another post).
http://www.bjcp.org/officers.php
Send a note to the Exam Directorate as indicated on the BJCP officers page link above and stated earlier in the thread.
Quote from: richardt
All I ask is that, if I spend the time in good faith creating and submitting these questions, that they are going to be seriously considered along with a format change towards an objective examination format.  Can you guarantee that?  If not, I'm a busy man and I'm not going to waste my time.  I'll be even more to the point--If you're asking me to join the BJCP exam directorate just to grade more written exams, my answer is a firm "No, thank you."
Yes, they are considering a format change for the entrance exam, but as a volunteer organization the wheels turn slowly without assistance. Everyone has advice, few offer assistance, true sweat equity!

Are you already grading exams? If so, good for you, if not, why type the word "more"?

Quote from: richardt
My point about the number of master judges (88) simply has to do with the fact that most Master Judges only score 90-92 on the exam.  Really?  Not one individual ever "aced" it?  Come on, ...Really?  There are people who "ace" the SAT's!  This tells me that the BJCP grading process and/or exam format is flawed.  Just like I perceive the scoring process to be at some beer competitions.  I'm not the only one--I didn't start this thread.

I once took a class in college where the instructor told us he would grade on a curve. My first exam was a 65 and when all was said and done my average for the class was a 60 yet I ended up with a high B. I have no problem with the BJCP exam and the grading. As far as an SAT analogy, about 20 out of 1,000,000 get a perfect score. We have not quite hit the million mark with the BJCP exam so it is coming.  ;)
Seriously, the problem is people don't answer the question posed even when they know where the points will be coming from. Take the gimme grid, T/F purposes of the BJCP. I've only ever graded a few exams that got them all correct and the answers are given in the Study Guide and on the website. I've given plenty of 10 out of 10 for individual questions, but inevitably the examinee falters on some other question.
It's not a popularity contest, it's beer!

Offline richardt

  • Senior Brewmaster
  • ******
  • Posts: 1227
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #37 on: July 11, 2010, 08:21:22 AM »
a format change towards an objective examination format.

I love the idea, but how would that work? Eliminate the tasting portion?

For the time allotted, I find the writing portion to be an inefficient way of assessing a person's knowledge and requires much subjectivity in assigning a grade.  Most of us can't write fast and legibly or with the gifted prose of a Michael Jackson, Charlie Papazian, or Garrett Oliver.  It is that portion that I would favor changing to a multiple choice type exam (you could easily ask 150 questions of varied levels of difficulty and replete with graphs, photos, diagrams, etc.).  Most professional society certification and licensing exams are in that format.  I'd imagine that portion could be more easily given at testing centers around the country (most city colleges have them)  and graded automatically/mechanically (unless it excessively drives up the cost of the exam)--it is worth looking into if it hasn't already been explored.

Sean, the sensory/tasting portion should be more robust, in my opinion.  After all, that is the big task we have as BJCP judges.  Obviously, we can't consume a ton of beers during the exam, but a lot of the off-aromas can be perceivd by smell alone, so why not have 5 or 10 "scratch-n-sniff" cards or plain "business cards" that get sprayed with the off-aroma (much like the perfume section of the local department store).  And the answers would be in multiple choice format or single word/sentence responses).  As it stands now, IMO we spend too much time in preparation for the written portion of the exam when more emphasis should be put on training our senses to detect these flaws.  I see the exam as being a two part exam.  Take the written one first.  If you pass it, i.e., a Recognized or higher score (60+), then go ahead and sign up to take the sensory exam.
We're often advised to take the exam multiple times and to take just one part and not the other, in order to have more time to finish and give good responses.

Online denny

  • Administrator
  • Retired with too much time on my hands
  • *****
  • Posts: 13686
  • Noti OR [1991.4, 287.6deg] AR
    • View Profile
    • Dennybrew
Re: A question about the BJCP Judging philosophy?
« Reply #38 on: July 11, 2010, 08:28:07 AM »
The "glass ceiling" or upper limit of 43-45 with BJCP scoring needs to be re-evaluated by the BJCP Board and consideration given towards issuing a position statement to all BJCP judges regarding the upper limit for scores in the BJCP guidelines.

IMO, a score of 50 should be possible to achieve.
I know of no other competition involving subjective judging of objective performances (e.g., diving, gymnastics, snow-boarding) that does not allow a "perfect" score to be given.
A brewer's 12 ounce bottled performance deserves the same chance.


Of course i should be possible to achieve.  The fact that I've never judged a beer that deserves a perfect score doesn't mean that there can't be one.

I also note how the BJCP exam itself is damn near impossible to break above 90, as well; especially given its format (written exam) and time limitations.  Over 5 thousand bright homebrewers have taken the exam and only 88 people "subjectively" scored in the 90's on the exam and met the experience criteria to become a "Master" BJCP judge.  The time has come to make the exam easier to take, easier to grade, and easier to learn how to be a good judge. 

I've taken the exam one time and got an 89.  Close enough to 90 to make the point that it's possible to do it.  Not to be snarky, but I don't see the pint in making the exam easier just so more people score higher.
Life begins at 60.....1.060, that is!

www.dennybrew.com

The best, sharpest, funniest, weirdest and most knowledgable minds in home brewing contribute on the AHA forum. - Alewyfe

Offline richardt

  • Senior Brewmaster
  • ******
  • Posts: 1227
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #39 on: July 11, 2010, 08:43:08 AM »
It may be easier to start a new organization than change the current one. Just sayin' ...  :-\

I'm definitely not looking to break up the BJCP--it think it has a noble goal and its members are good people.
I'm just trying to suggest ways that the BJCP and its members can accomplish the mission better and more efficiently.
The current exam format is stodgy.  It is time to improve it.  It is a major gripe of those who consider taking and take the exam.

The sensory expertise of BJCP judges should be honed at every beer club meeting.  This can come in the form of commercial calibration sessions at each meeting, for example.  I also think every homebrew club should get at least one "off-flavors and aromas" kit from Siebel (free from BJCP?) to keep the club membership's expertise current.  Participation in one of those events with your homebrew club should count towards BJCP experience points.

All these efforts should help reduce the instances where two judges would have vastly disparate scores that need reconcilliation.

Offline richardt

  • Senior Brewmaster
  • ******
  • Posts: 1227
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #40 on: July 11, 2010, 09:03:06 AM »
Quote from: richardt
All I ask is that, if I spend the time in good faith creating and submitting these questions, that they are going to be seriously considered along with a format change towards an objective examination format.  Can you guarantee that?  If not, I'm a busy man and I'm not going to waste my time.  I'll be even more to the point--If you're asking me to join the BJCP exam directorate just to grade more written exams, my answer is a firm "No, thank you."
Quote
Yes, they are considering a format change for the entrance exam, but as a volunteer organization the wheels turn slowly without assistance. Everyone has advice, few offer assistance, true sweat equity!

Are you already grading exams? If so, good for you, if not, why type the word "more"?

I'll contact the BJCP shortly to offer my assistance and suggest MC exam questions.  I don't grade BJCP exams and I haven't been asked to do so.  However, if there's no serious effort or interest to convert to a multiple choice format and improve the overall experience for everyone, then I'm not going to get involved and offer my sweat just to help grade more written BJCP exams and maintain the status quo.  The grading process is a significant bottleneck--BJCP exams are currently limited to 8 exams/month and 12 examinees/exam, unless special permission is granted to expand up to 20 examinees.


Offline richardt

  • Senior Brewmaster
  • ******
  • Posts: 1227
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #41 on: July 11, 2010, 09:41:35 AM »
Of course i should be possible to achieve.  The fact that I've never judged a beer that deserves a perfect score doesn't mean that there can't be one.

I've taken the exam one time and got an 89.  Close enough to 90 to make the point that it's possible to do it.  Not to be snarky, but I don't see the pint in making the exam easier just so more people score higher.

So, bottom-line, a 13 is just a courtesy score for a really bad beer, and, theoretically, a beer can score 49-50 if it is truly outstanding!  I'm fine with that.  I have tasted some that I'd been happy to award a score in the high 40's (unfortunately, none of them are my own--I'm working on that). 

Denny, good job on the 89--that is a great score.  I chalk my exam results up to my relative sensory inexperience/lack of training for the sensory exam and poor time management on the written exam.  I will be taking the exam again next year when our club is due to retake it.  I know you're not being snarky.  I've taken many MC exams that really explore my knowledge base and just leave me feeliing rattled no matter how well I've prepared for it.  By "easier," I mean to take and grade the exams, not necessarily to score higher on the exams.

Pyrite, sorry about the thread hijack.  Good luck on studying for your BJCP exam.  Research the answers to the questions and write them out beforehand.  I suggest you keep written answers to 100-120 words max.  Make them meaty and concise and easy to remember/regurgitate on the exam (I know... easy to say, hard to do).  The graders also seem to like it when you spit out the beer style parameters (OG, FG, ABV, IBU's, SRM).
« Last Edit: July 11, 2010, 09:55:49 AM by richardt »

Offline brewballs

  • Cellarman
  • **
  • Posts: 42
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #42 on: July 11, 2010, 10:06:07 AM »
So much stress, when home brew is meant to be enjoying and relaxing.

Just makes me want to say, RDWHAHB.
Change it back.

Offline MDixon

  • Senior Brewmaster
  • ******
  • Posts: 1241
    • View Profile
    • Mike's Homebrewing Page
Re: A question about the BJCP Judging philosophy?
« Reply #43 on: July 11, 2010, 04:26:20 PM »
For the record, I have been trying to be snarky, if it didn't come off as such, I apologize for my poor effort  ;)

I believe the thought behind a revised exam for entry levels is M/C & T/F for the written which may comprise ~50% of the effort. The taste would be increased to perhaps 6 beers and would comprise ~50% of the exam.

My thinking has been to have an online session which is timed which must be passed to even take the entry level exam. Then the exam as described above. Finally a standard exam for the higher ranks. I'm unsure, but it could be the standard exam may be revised to be a 50/50 exam with 6 beer evaluations.

FWIW - the problem with sensory additions or samples is freshness and how well they are mixed and used. I've seen where incomplete mixing caused one proctor to get little of the off component in their glass and another proctor to end up with a beer which was near impossible to choke down. Too much variability with the methods meant to eliminate variability. I'd much prefer three proctors and the examinees.
It's not a popularity contest, it's beer!

Offline micsager

  • Senior Brewmaster
  • ******
  • Posts: 1142
    • View Profile
Re: A question about the BJCP Judging philosophy?
« Reply #44 on: July 11, 2010, 07:53:32 PM »
Lot's of opinions on this subject, for sure.  I helped stewart one competition, and have entered two.  I enjoyed the feedback on both I entered, and it was interesting to see how a comp is run by stewarting. 

All I will say is the folks that do judging, and run the competitions are terrific.  They get nothing out of it, other than the good feeling of being a home brewer. 

If you don't like the system, get involved, and work to change it from the inside.  Like many have said, it's easy to whine about what's f'ed up.