Author Topic: Interesting Judging Phenomenon  (Read 950 times)

Offline ynotbrusum

  • Senior Brewmaster
  • ******
  • Posts: 1517
  • Da mihi sis cerevisiam.
    • View Profile
Interesting Judging Phenomenon
« on: January 30, 2014, 07:55:11 PM »
Just watched an episode of Brain Games an saw a test that struck me as very interesting - they had two panels examine a set of products.  The women's panel examined and rated 5 nylon stockings and the men's panel examined and rated 5 pairs of blue jeans - both were using a given criteria set by which to judge the items.

Interestingly, the items were identical!  But the judges were not told that they were the same and rated them differently.  It made me wonder if a panel of BJCP judges were set up the same way, how well they would do analyzing the same beer?

When told about the test some of the subjects then acknowledged that they thought they were the same, while others still felt that there were legitimate differences.  The point is, if you expect there to be discernable differences, your brain can trick you into finding them...

Something to think about.
Hodge Garage Brewing: "Brew with a glad heart!"

Offline duxx

  • Cellarman
  • **
  • Posts: 34
  • Good Karma Brewing, Warrensburg, MO
    • View Profile
Re: Interesting Judging Phenomenon
« Reply #1 on: January 31, 2014, 07:26:27 AM »
Almost everyone who has entered the same beer in multiple competitions has a story about how their beer got a 40 in one competition and a 20 in a different competition a short time later.  So I suppose it is certainly possible.
"Tan and lean like a longneck bottle."  Zac Brown Band.

Offline Jimmy K

  • Official Poobah of No Life.
  • *
  • Posts: 3509
  • Delaware
    • View Profile
Re: Interesting Judging Phenomenon
« Reply #2 on: January 31, 2014, 07:42:10 AM »
I think this is more like the "I don't want to look stupid" effect. You're given 5 identical items and told to rate them. If you say they are identical, you run the risk of being wrong and showing that you can't discern between them. Then you look stupid. Since you don't want to look stupid, you "find" differences.
 
I think the 'finding differences' is sub-conscious. But you have to be very confident in your own abilities to tell the person organizing that they are wrong.
« Last Edit: January 31, 2014, 08:43:33 AM by mtnrockhopper »
Delmarva United Homebrewers - President by inverse coup - former president ousted himself.
AHA Member since 2006
BJCP Certified: B0958

Offline morticaixavier

  • Official Poobah of No Life.
  • *
  • Posts: 7111
  • Davis, CA
    • View Profile
    • The Best Artist in the WORLD!!!!!
Re: Interesting Judging Phenomenon
« Reply #3 on: January 31, 2014, 08:56:26 AM »
this is why a proper triangle test has two stages.

first you eliminate those judges who simply get it wrong (say all are the same, all are different when two are the same etc.) and then use the feedback from the judges who got the differences right.

Offline james

  • Brewer
  • ****
  • Posts: 361
    • View Profile
Re: Interesting Judging Phenomenon
« Reply #4 on: January 31, 2014, 09:38:23 AM »
One time I entered a Munich helles as itself and a blonde ale, the helles took BOS and the blonde took honorable mention in BOS.  The judges I talked to swore they tasted different.  I bottled them at the exact same time, same keg, etc.

Offline denny

  • Administrator
  • Retired with too much time on my hands
  • *****
  • Posts: 13307
  • Noti OR [1991.4, 287.6deg] AR
    • View Profile
    • Dennybrew
Re: Interesting Judging Phenomenon
« Reply #5 on: January 31, 2014, 09:47:38 AM »
There's an entire chapter in "Experimental Brewing" dedicated to evaluation, and we examine this phenomenon in some depth.  This experiment has been repeated many times with the same results.
Life begins at 60.....1.060, that is!

www.dennybrew.com

The best, sharpest, funniest, weirdest and most knowledgable minds in home brewing contribute on the AHA forum. - Alewyfe

Offline dbeechum

  • Administrator
  • Brewmaster General
  • *****
  • Posts: 2395
  • Pasadena, CA
    • View Profile
    • Experimental Brewing
Re: Interesting Judging Phenomenon
« Reply #6 on: January 31, 2014, 09:53:17 AM »
Yup.. human perception is remarkably susceptible to suggestion. It's the gordian knot of subjective evaluation.
Drew Beechum - Maltosefalcons.com
- Vote in the AHA GC Election! - http://bit.ly/1aV9GVd  -
-----
Burbling:
Gnome is in the Details
*Experimental Brewing - The Book*
Tap:
Peanut Butter Jelly Time
Tupelo Mead
Farmhouse Brett Saison

Offline morticaixavier

  • Official Poobah of No Life.
  • *
  • Posts: 7111
  • Davis, CA
    • View Profile
    • The Best Artist in the WORLD!!!!!
Re: Interesting Judging Phenomenon
« Reply #7 on: January 31, 2014, 09:57:26 AM »
There is also the experiment where they dyed white wine red with a flavourless, odorless dye and tasters were completely taken in.

Offline hopfenundmalz

  • Official Poobah of No Life.
  • *
  • Posts: 5671
  • Milford, MI
    • View Profile
Re: Interesting Judging Phenomenon
« Reply #8 on: January 31, 2014, 10:40:09 AM »
One time I entered a Munich helles as itself and a blonde ale, the helles took BOS and the blonde took honorable mention in BOS.  The judges I talked to swore they tasted different.  I bottled them at the exact same time, same keg, etc.
A friend did that, entered the same beer as a Dortmunder and German Pils, both medaled.
Jeff Rankert
Ann Arbor Brewers Guild, AHA Member, BJCP Certified
Home-brewing, not just a hobby, it is a lifestyle!

Offline AmandaK

  • Senior Brewmaster
  • ******
  • Posts: 1235
  • Redbird Brewhouse
    • View Profile
Re: Interesting Judging Phenomenon
« Reply #9 on: January 31, 2014, 11:26:53 AM »
One time I entered a Munich helles as itself and a blonde ale, the helles took BOS and the blonde took honorable mention in BOS.  The judges I talked to swore they tasted different.  I bottled them at the exact same time, same keg, etc.
A friend did that, entered the same beer as a Dortmunder and German Pils, both medaled.

I do that with my lambics. It's always a crap-shoot as to if the lambic entered as a lambic will win or the lambic entered as a gueuze will win. I haven't found a pattern yet, and I don't think I will. I will be entering a different "trick beer" into the KCBM XXXI Competition, same beer, two categories - just to see what happens.  :)
Amanda Burkemper
Kansas City Bier Meister
BJCP Master

Redbird Brewhouse - Finishing Touches on the Basement Bar

Offline udubdawg

  • Brewmaster
  • *****
  • Posts: 743
    • View Profile
Re: Interesting Judging Phenomenon
« Reply #10 on: January 31, 2014, 11:54:08 AM »
There is also the experiment where they dyed white wine red with a flavourless, odorless dye and tasters were completely taken in.

I did this with an IPA.  amazing how prominent the "roasty" qualities suddenly became to the judges.   ::)

Offline denny

  • Administrator
  • Retired with too much time on my hands
  • *****
  • Posts: 13307
  • Noti OR [1991.4, 287.6deg] AR
    • View Profile
    • Dennybrew
Re: Interesting Judging Phenomenon
« Reply #11 on: January 31, 2014, 11:54:54 AM »
There is also the experiment where they dyed white wine red with a flavourless, odorless dye and tasters were completely taken in.

That one is in the book!
Life begins at 60.....1.060, that is!

www.dennybrew.com

The best, sharpest, funniest, weirdest and most knowledgable minds in home brewing contribute on the AHA forum. - Alewyfe

Online dmtaylor

  • Senior Brewmaster
  • ******
  • Posts: 1256
  • Two Rivers, WI
    • View Profile
Re: Interesting Judging Phenomenon
« Reply #12 on: January 31, 2014, 11:57:32 AM »
I've had one of my beers tasted twice by the same judge in two different competitions just a few weeks apart, with two completely different scores.  I believe it was a Baltic porter or an old ale or something like that.  The first time he gave it mid-30s, and the second time a 17.  Also worth noting, the first time he was judging with a good reasonable judge, and the second time with an arrogant butthole who gave it a 13.  I am a judge myself so I knew that my beer deserved a score in the 30s.  I hate buttholes.  Think they can influence the world in any way they see fit even when they are clearly wrong.  But of course it's also the other guy's fault for letting himself be influenced by a butthole.  At least I know his true opinion.

It's also interesting judging your own beer blind in a competition or whatever when you don't know that it's your own.  With your bias aside, you can get some really great opinions from your own self on your own beer.  Impossible when you know that it's your own.  Pretty cool stuff.
Dave

"This is grain, which any fool can eat, but for which the Lord intended a more divine means of consumption. Let us give praise to our Maker, and glory to His bounty, by learning about... BEER!" - Friar Tuck (Robin Hood - Prince of Thieves)

Offline ccfoo242

  • Brewmaster
  • *****
  • Posts: 778
  • I drank what? - Socrates
    • View Profile
Re: Interesting Judging Phenomenon
« Reply #13 on: January 31, 2014, 12:55:33 PM »
In two different comps I had an odd thing happen with the same entry: one judge said no hop aroma, the other said too much hop aroma in each comp. Just can't please some people.


Regarding judging your own beer blind a few months ago I had an entry with a wild yeast infection. It wasn't until I tasted it thinking I was having someone else's beer that I noticed how bad it was.


Perception is a weird thing.

Isn't there a bjcp rule about the lowest score being 15?  13 seems just mean.


Sent from the future using Tapatalk

Intra cervisiam est deus.

Offline Jimmy K

  • Official Poobah of No Life.
  • *
  • Posts: 3509
  • Delaware
    • View Profile
Re: Interesting Judging Phenomenon
« Reply #14 on: January 31, 2014, 01:07:27 PM »
Isn't there a bjcp rule about the lowest score being 15?  13 seems just mean.
13 is BJCP's low, though anything in the teens must be terrible and I think many competitions/judges don't go below 17 or 19. I mean once you're that low, the actual score doesn't matter anyway.
 
A great prank would be to send somebody scoresheets with a score under 10.
Delmarva United Homebrewers - President by inverse coup - former president ousted himself.
AHA Member since 2006
BJCP Certified: B0958