Author Topic: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance  (Read 2639 times)

Offline brulosopher

  • Brewmaster
  • *****
  • Posts: 505
  • They who drink beer will think beer
    • View Profile
    • Brülosophy
The rate of statistically significant exBEERiment findings currently hovers around 35%, a respectable number in my opinion, though far lower than many have expected given the presumable importance of the variables investigated. While it's only natural to come up with excuses as to how this could be, none have been more popular than what I've come to call... The s***ty Palates Argument. Greg and I sought the assistance of bloggers Scott Janish and Justin Angevaare to help us compare triangle test performance rates based on level of experience. Are BJCP judges actually better at distinguishing differences than general beer drinkers? Read to find out!

http://brulosophy.com/2016/01/21/investigating-the-bad-palates-argument-a-graphical-look-at-xbmt-performance-based-on-experience-level/

Offline AmandaK

  • Senior Brewmaster
  • ******
  • Posts: 1850
  • Redbird Brewhouse
    • View Profile
The rate of statistically significant exBEERiment findings currently hovers around 35%, a respectable number in my opinion, though far lower than many have expected given the presumable importance of the variables investigated. While it's only natural to come up with excuses as to how this could be, none have been more popular than what I've come to call... The s***ty Palates Argument. Greg and I sought the assistance of bloggers Scott Janish and Justin Angevaare to help us compare triangle test performance rates based on level of experience. Are BJCP judges actually better at distinguishing differences than general beer drinkers? Read to find out!

http://brulosophy.com/2016/01/21/investigating-the-bad-palates-argument-a-graphical-look-at-xbmt-performance-based-on-experience-level/

Side bar: I love that you got Janish in on this.

Interesting data point that wasn't touched on in the article: the % correct responses for "provisional" judges was a full 20% lower than everyone else. Any thoughts on why that is? Lower sample size or something more intriguing?
Amanda Burkemper
KC Bier Meisters Education Director
BJCP Assistant Education Director
BJCP Master/Mead

Redbird Brewhouse - There's Always a Project
Our Homebrewed Wedding, AHA Article

Offline brulosopher

  • Brewmaster
  • *****
  • Posts: 505
  • They who drink beer will think beer
    • View Profile
    • Brülosophy
The rate of statistically significant exBEERiment findings currently hovers around 35%, a respectable number in my opinion, though far lower than many have expected given the presumable importance of the variables investigated. While it's only natural to come up with excuses as to how this could be, none have been more popular than what I've come to call... The s***ty Palates Argument. Greg and I sought the assistance of bloggers Scott Janish and Justin Angevaare to help us compare triangle test performance rates based on level of experience. Are BJCP judges actually better at distinguishing differences than general beer drinkers? Read to find out!

http://brulosophy.com/2016/01/21/investigating-the-bad-palates-argument-a-graphical-look-at-xbmt-performance-based-on-experience-level/

Side bar: I love that you got Janish in on this.

Interesting data point that wasn't touched on in the article: the % correct responses for "provisional" judges was a full 20% lower than everyone else. Any thoughts on why that is? Lower sample size or something more intriguing?

Scott is a badass!

I noticed that odd stat as well and honestly have no clue why it occurred. What i find most curious is the fact what you pointed out is only true when looking at only significant xBmts, as provisional judges performed about the same as recognized and higher on all xBmts combined. Anomaly perhaps?

Offline klickitat jim

  • I must live here
  • **********
  • Posts: 8460
    • View Profile
If you assume that most or all BJCP judges in training are craft beer enthusiasts and home brewers, accounting for why they suddenly fall way short of their otherwise contemporaries could be explained by understanding how people learn a new skill. Frequently, while assimilating new knowledge, people drop dramatically in ability while they are struggling to get this new info to fit into their world.

I have taught a certain set of psychmotor skills to professionals in my day job. At the beginning of the session, if you have them perform an open skills test, using their existing skill level, they do fairly well. Then teach them several new skills for a few hours. Then go back to the original test, trying to use the new skills. They almost always perform worse than they did before the learning. Thats why its a futile plan to teach too much too soon when it comes to psychomotor skills. Analytically tasting beer is a psychomotor skill. Its hard for new judges to get out of their head. It makes perfect sense to me that a craft beer enthusiast would do fairly well, an experienced home brewery slightly better, and an experienced judge slightly better yet, and the judge in training worst of all. Give them time, they will be at the top.

Offline Phil_M

  • Senior Brewmaster
  • ******
  • Posts: 1689
  • Southern Maryland
    • View Profile
I agree with Jim, and also have seen that sort of thing happen at work. They're more likely to overthink the issue, and subsequently doubt their initial response a bit more.

This data just seems reasonable to me. It makes sense that on average, BJCP judges have palates that are, well, average.

I'd hope this would encourage folks with the time to study to become BJCP judges, especially if they've ever felt that their palates aren't good enough.
Corn is a fine adjunct in beer.

And don't buy stale beer.

Offline brulosopher

  • Brewmaster
  • *****
  • Posts: 505
  • They who drink beer will think beer
    • View Profile
    • Brülosophy
If you assume that most or all BJCP judges in training are craft beer enthusiasts and home brewers, accounting for why they suddenly fall way short of their otherwise contemporaries could be explained by understanding how people learn a new skill. Frequently, while assimilating new knowledge, people drop dramatically in ability while they are struggling to get this new info to fit into their world.

I have taught a certain set of psychmotor skills to professionals in my day job. At the beginning of the session, if you have them perform an open skills test, using their existing skill level, they do fairly well. Then teach them several new skills for a few hours. Then go back to the original test, trying to use the new skills. They almost always perform worse than they did before the learning. Thats why its a futile plan to teach too much too soon when it comes to psychomotor skills. Analytically tasting beer is a psychomotor skill. Its hard for new judges to get out of their head. It makes perfect sense to me that a craft beer enthusiast would do fairly well, an experienced home brewery slightly better, and an experienced judge slightly better yet, and the judge in training worst of all. Give them time, they will be at the top.

Something to note-- many of the provisional judges from a year ago are now recognized or certified judges today, and the provisional judges today will likely be recognized or certified a year from now. I'm not convinced their performance was hindered by overthinking at all, but rather the fact any differences between the samples simply weren't noticeable. I'm biased, of course, as I taste a large majority of the xBmt beers and triangle myself every time, it's not as easy as people think... it's not as easy as I thought it would be.

Offline dmtaylor

  • Official Poobah of No Life.
  • *
  • Posts: 3224
  • Two Rivers, WI
    • View Profile
Interesting data point that wasn't touched on in the article: the % correct responses for "provisional" judges was a full 20% lower than everyone else.

This is very interesting.

Nice work on the xbmt -- it needed to be done.  Pretty clear results.
Dave

The world will become a much more pleasant place to live when each and every one of us realizes that we are all idiots.

Offline HoosierBrew

  • Global Moderator
  • I must live here
  • *****
  • Posts: 13030
  • Indianapolis,IN
    • View Profile
All in all, seems pretty reasonable. Thanks, Marshall !
Jon H.

Offline charles1968

  • Brewmaster
  • *****
  • Posts: 532
    • View Profile
Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

Offline brulosopher

  • Brewmaster
  • *****
  • Posts: 505
  • They who drink beer will think beer
    • View Profile
    • Brülosophy

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

Offline AmandaK

  • Senior Brewmaster
  • ******
  • Posts: 1850
  • Redbird Brewhouse
    • View Profile

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

I did two tastings at NHC. Got 50% wrong.  ;)
Amanda Burkemper
KC Bier Meisters Education Director
BJCP Assistant Education Director
BJCP Master/Mead

Redbird Brewhouse - There's Always a Project
Our Homebrewed Wedding, AHA Article

Offline brulosopher

  • Brewmaster
  • *****
  • Posts: 505
  • They who drink beer will think beer
    • View Profile
    • Brülosophy

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

I did two tastings at NHC. Got 50% wrong.  ;)

I prefer to look at it as you getting half right! Haha.

Offline AmandaK

  • Senior Brewmaster
  • ******
  • Posts: 1850
  • Redbird Brewhouse
    • View Profile

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

I did two tastings at NHC. Got 50% wrong.  ;)

I prefer to look at it as you getting half right! Haha.
Where either of those significant? I think I remember no, which means I'm some measure better than average!
Amanda Burkemper
KC Bier Meisters Education Director
BJCP Assistant Education Director
BJCP Master/Mead

Redbird Brewhouse - There's Always a Project
Our Homebrewed Wedding, AHA Article

Offline brulosopher

  • Brewmaster
  • *****
  • Posts: 505
  • They who drink beer will think beer
    • View Profile
    • Brülosophy


Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

I did two tastings at NHC. Got 50% wrong.  ;)

I prefer to look at it as you getting half right! Haha.
Where either of those significant? I think I remember no, which means I'm some measure better than average!

Not even close to significant.

Offline jeffy

  • Official Poobah of No Life.
  • *
  • Posts: 3392
  • Tampa, Fl
    • View Profile


Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

I did two tastings at NHC. Got 50% wrong.  ;)

I prefer to look at it as you getting half right! Haha.
Where either of those significant? I think I remember no, which means I'm some measure better than average!

Not even close to significant.
Well, I got 'em both right and I couldn't smell anything.  Does that make me insignificant?
Jeff Gladish, Tampa (989.3, 175.1 Apparent Rennarian)
Homebrewing since 1990
AHA member since 1991, now a lifetime member
BJCP judge since 1995