Membership questions? Log in issues? Email info@brewersassociation.org

Author Topic: Interesting Judging Phenomenon  (Read 3392 times)

Offline ynotbrusum

  • Official Poobah of No Life. (I Got Ban Hammered by Drew)
  • *********
  • Posts: 4874
Interesting Judging Phenomenon
« on: January 30, 2014, 07:55:11 pm »
Just watched an episode of Brain Games an saw a test that struck me as very interesting - they had two panels examine a set of products.  The women's panel examined and rated 5 nylon stockings and the men's panel examined and rated 5 pairs of blue jeans - both were using a given criteria set by which to judge the items.

Interestingly, the items were identical!  But the judges were not told that they were the same and rated them differently.  It made me wonder if a panel of BJCP judges were set up the same way, how well they would do analyzing the same beer?

When told about the test some of the subjects then acknowledged that they thought they were the same, while others still felt that there were legitimate differences.  The point is, if you expect there to be discernable differences, your brain can trick you into finding them...

Something to think about.
Hodge Garage Brewing: "Brew with a glad heart!"

Offline duxx

  • Cellarman
  • **
  • Posts: 37
  • Good Karma Brewing, Warrensburg, MO
Re: Interesting Judging Phenomenon
« Reply #1 on: January 31, 2014, 07:26:27 am »
Almost everyone who has entered the same beer in multiple competitions has a story about how their beer got a 40 in one competition and a 20 in a different competition a short time later.  So I suppose it is certainly possible.
"Tan and lean like a longneck bottle."  Zac Brown Band.

Offline Jimmy K

  • Official Poobah of No Life. (I Got Ban Hammered by Drew)
  • *********
  • Posts: 3643
  • Delaware
Re: Interesting Judging Phenomenon
« Reply #2 on: January 31, 2014, 07:42:10 am »
I think this is more like the "I don't want to look stupid" effect. You're given 5 identical items and told to rate them. If you say they are identical, you run the risk of being wrong and showing that you can't discern between them. Then you look stupid. Since you don't want to look stupid, you "find" differences.
 
I think the 'finding differences' is sub-conscious. But you have to be very confident in your own abilities to tell the person organizing that they are wrong.
« Last Edit: January 31, 2014, 08:43:33 am by mtnrockhopper »
Delmarva United Homebrewers - President by inverse coup - former president ousted himself.
AHA Member since 2006
BJCP Certified: B0958

Offline morticaixavier

  • I must live here
  • **********
  • Posts: 7781
  • Underhill VT
    • The Best Artist in the WORLD!!!!!
Re: Interesting Judging Phenomenon
« Reply #3 on: January 31, 2014, 08:56:26 am »
this is why a proper triangle test has two stages.

first you eliminate those judges who simply get it wrong (say all are the same, all are different when two are the same etc.) and then use the feedback from the judges who got the differences right.
"Creativity is the residue of wasted time"
-A Einstein

"errors are [...] the portals of discovery"
- J Joyce

Offline james

  • Brewer
  • ****
  • Posts: 361
Re: Interesting Judging Phenomenon
« Reply #4 on: January 31, 2014, 09:38:23 am »
One time I entered a Munich helles as itself and a blonde ale, the helles took BOS and the blonde took honorable mention in BOS.  The judges I talked to swore they tasted different.  I bottled them at the exact same time, same keg, etc.

Offline denny

  • Administrator
  • Retired with too much time on my hands
  • *****
  • Posts: 27070
  • Noti OR [1991.4, 287.6deg] AR
    • Dennybrew
Re: Interesting Judging Phenomenon
« Reply #5 on: January 31, 2014, 09:47:38 am »
There's an entire chapter in "Experimental Brewing" dedicated to evaluation, and we examine this phenomenon in some depth.  This experiment has been repeated many times with the same results.
Life begins at 60.....1.060, that is!

www.dennybrew.com

The best, sharpest, funniest, weirdest and most knowledgable minds in home brewing contribute on the AHA forum. - Alewyfe

"The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts." - Bertrand Russell

Offline dbeechum

  • Global Moderator
  • I spend way too much time on the AHA forum
  • *****
  • Posts: 2913
  • Pasadena, CA
    • Experimental Brewing
Re: Interesting Judging Phenomenon
« Reply #6 on: January 31, 2014, 09:53:17 am »
Yup.. human perception is remarkably susceptible to suggestion. It's the gordian knot of subjective evaluation.
Drew Beechum - Maltosefalcons.com
- Vote in the AHA GC Election! - http://bit.ly/1aV9GVd  -
-----
Burbling:
Gnome is in the Details
*Experimental Brewing - The Book*
Tap:
Peanut Butter Jelly Time
Tupelo Mead
Farmhouse Brett Saison

Offline morticaixavier

  • I must live here
  • **********
  • Posts: 7781
  • Underhill VT
    • The Best Artist in the WORLD!!!!!
Re: Interesting Judging Phenomenon
« Reply #7 on: January 31, 2014, 09:57:26 am »
There is also the experiment where they dyed white wine red with a flavourless, odorless dye and tasters were completely taken in.
"Creativity is the residue of wasted time"
-A Einstein

"errors are [...] the portals of discovery"
- J Joyce

Offline hopfenundmalz

  • Global Moderator
  • I must live here
  • *****
  • Posts: 10676
  • Milford, MI
Re: Interesting Judging Phenomenon
« Reply #8 on: January 31, 2014, 10:40:09 am »
One time I entered a Munich helles as itself and a blonde ale, the helles took BOS and the blonde took honorable mention in BOS.  The judges I talked to swore they tasted different.  I bottled them at the exact same time, same keg, etc.
A friend did that, entered the same beer as a Dortmunder and German Pils, both medaled.
Jeff Rankert
AHA Lifetime Member
BJCP National
Ann Arbor Brewers Guild
Home-brewing, not just a hobby, it is a lifestyle!

Offline AmandaK

  • Senior Brewmaster
  • ******
  • Posts: 1850
  • Redbird Brewhouse
Re: Interesting Judging Phenomenon
« Reply #9 on: January 31, 2014, 11:26:53 am »
One time I entered a Munich helles as itself and a blonde ale, the helles took BOS and the blonde took honorable mention in BOS.  The judges I talked to swore they tasted different.  I bottled them at the exact same time, same keg, etc.
A friend did that, entered the same beer as a Dortmunder and German Pils, both medaled.

I do that with my lambics. It's always a crap-shoot as to if the lambic entered as a lambic will win or the lambic entered as a gueuze will win. I haven't found a pattern yet, and I don't think I will. I will be entering a different "trick beer" into the KCBM XXXI Competition, same beer, two categories - just to see what happens.  :)
Amanda Burkemper
KC Bier Meisters Lifetime Member - KCBM 3x AHA Club of the Year!!
BJCP Assistant (to the) Midwest Rep
BJCP Grand Master/Mead/Cider


Our Homebrewed Wedding, AHA Article

Offline udubdawg

  • Senior Brewmaster
  • ******
  • Posts: 1081
Re: Interesting Judging Phenomenon
« Reply #10 on: January 31, 2014, 11:54:08 am »
There is also the experiment where they dyed white wine red with a flavourless, odorless dye and tasters were completely taken in.

I did this with an IPA.  amazing how prominent the "roasty" qualities suddenly became to the judges.   ::)

Offline denny

  • Administrator
  • Retired with too much time on my hands
  • *****
  • Posts: 27070
  • Noti OR [1991.4, 287.6deg] AR
    • Dennybrew
Re: Interesting Judging Phenomenon
« Reply #11 on: January 31, 2014, 11:54:54 am »
There is also the experiment where they dyed white wine red with a flavourless, odorless dye and tasters were completely taken in.

That one is in the book!
Life begins at 60.....1.060, that is!

www.dennybrew.com

The best, sharpest, funniest, weirdest and most knowledgable minds in home brewing contribute on the AHA forum. - Alewyfe

"The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts." - Bertrand Russell

Online dmtaylor

  • Official Poobah of No Life. (I Got Ban Hammered by Drew)
  • *********
  • Posts: 4716
  • Lord Idiot the Lazy
    • YEAST MASTER Perma-Living
Re: Interesting Judging Phenomenon
« Reply #12 on: January 31, 2014, 11:57:32 am »
I've had one of my beers tasted twice by the same judge in two different competitions just a few weeks apart, with two completely different scores.  I believe it was a Baltic porter or an old ale or something like that.  The first time he gave it mid-30s, and the second time a 17.  Also worth noting, the first time he was judging with a good reasonable judge, and the second time with an arrogant butthole who gave it a 13.  I am a judge myself so I knew that my beer deserved a score in the 30s.  I hate buttholes.  Think they can influence the world in any way they see fit even when they are clearly wrong.  But of course it's also the other guy's fault for letting himself be influenced by a butthole.  At least I know his true opinion.

It's also interesting judging your own beer blind in a competition or whatever when you don't know that it's your own.  With your bias aside, you can get some really great opinions from your own self on your own beer.  Impossible when you know that it's your own.  Pretty cool stuff.
Dave

The world will become a much more pleasant place to live when each and every one of us realizes that we are all idiots.

Offline ccfoo242

  • Brewmaster
  • *****
  • Posts: 808
  • I drank what? - Socrates
Re: Interesting Judging Phenomenon
« Reply #13 on: January 31, 2014, 12:55:33 pm »
In two different comps I had an odd thing happen with the same entry: one judge said no hop aroma, the other said too much hop aroma in each comp. Just can't please some people.


Regarding judging your own beer blind a few months ago I had an entry with a wild yeast infection. It wasn't until I tasted it thinking I was having someone else's beer that I noticed how bad it was.


Perception is a weird thing.

Isn't there a bjcp rule about the lowest score being 15?  13 seems just mean.


Sent from the future using Tapatalk
Intra cervisiam est deus.

Offline Jimmy K

  • Official Poobah of No Life. (I Got Ban Hammered by Drew)
  • *********
  • Posts: 3643
  • Delaware
Re: Interesting Judging Phenomenon
« Reply #14 on: January 31, 2014, 01:07:27 pm »
Isn't there a bjcp rule about the lowest score being 15?  13 seems just mean.
13 is BJCP's low, though anything in the teens must be terrible and I think many competitions/judges don't go below 17 or 19. I mean once you're that low, the actual score doesn't matter anyway.
 
A great prank would be to send somebody scoresheets with a score under 10.
Delmarva United Homebrewers - President by inverse coup - former president ousted himself.
AHA Member since 2006
BJCP Certified: B0958