[brulosopher]

The rate of statistically significant exBEERiment findings currently hovers around 35%, a respectable number in my opinion, though far lower than many have expected given the presumable importance of the variables investigated. While it's only natural to come up with excuses as to how this could be, none have been more popular than what I've come to call... The s***ty Palates Argument. Greg and I sought the assistance of bloggers Scott Janish and Justin Angevaare to help us compare triangle test performance rates based on level of experience. Are BJCP judges actually better at distinguishing differences than general beer drinkers? Read to find out!

http://brulosophy.com/2016/01/21/investigating-the-bad-palates-argument-a-graphical-look-at-xbmt-performance-based-on-experience-level/

[AmandaK]

The rate of statistically significant exBEERiment findings currently hovers around 35%, a respectable number in my opinion, though far lower than many have expected given the presumable importance of the variables investigated. While it's only natural to come up with excuses as to how this could be, none have been more popular than what I've come to call... The s***ty Palates Argument. Greg and I sought the assistance of bloggers Scott Janish and Justin Angevaare to help us compare triangle test performance rates based on level of experience. Are BJCP judges actually better at distinguishing differences than general beer drinkers? Read to find out!

http://brulosophy.com/2016/01/21/investigating-the-bad-palates-argument-a-graphical-look-at-xbmt-performance-based-on-experience-level/

Side bar: I love that you got Janish in on this.

Interesting data point that wasn't touched on in the article: the % correct responses for "provisional" judges was a full 20% lower than everyone else. Any thoughts on why that is? Lower sample size or something more intriguing?

[brulosopher]

The rate of statistically significant exBEERiment findings currently hovers around 35%, a respectable number in my opinion, though far lower than many have expected given the presumable importance of the variables investigated. While it's only natural to come up with excuses as to how this could be, none have been more popular than what I've come to call... The s***ty Palates Argument. Greg and I sought the assistance of bloggers Scott Janish and Justin Angevaare to help us compare triangle test performance rates based on level of experience. Are BJCP judges actually better at distinguishing differences than general beer drinkers? Read to find out!

http://brulosophy.com/2016/01/21/investigating-the-bad-palates-argument-a-graphical-look-at-xbmt-performance-based-on-experience-level/

Side bar: I love that you got Janish in on this.

Interesting data point that wasn't touched on in the article: the % correct responses for "provisional" judges was a full 20% lower than everyone else. Any thoughts on why that is? Lower sample size or something more intriguing?

Scott is a badass!

I noticed that odd stat as well and honestly have no clue why it occurred. What i find most curious is the fact what you pointed out is only true when looking at only significant xBmts, as provisional judges performed about the same as recognized and higher on all xBmts combined. Anomaly perhaps?

[klickitat jim]

If you assume that most or all BJCP judges in training are craft beer enthusiasts and home brewers, accounting for why they suddenly fall way short of their otherwise contemporaries could be explained by understanding how people learn a new skill. Frequently, while assimilating new knowledge, people drop dramatically in ability while they are struggling to get this new info to fit into their world.

I have taught a certain set of psychmotor skills to professionals in my day job. At the beginning of the session, if you have them perform an open skills test, using their existing skill level, they do fairly well. Then teach them several new skills for a few hours. Then go back to the original test, trying to use the new skills. They almost always perform worse than they did before the learning. Thats why its a futile plan to teach too much too soon when it comes to psychomotor skills. Analytically tasting beer is a psychomotor skill. Its hard for new judges to get out of their head. It makes perfect sense to me that a craft beer enthusiast would do fairly well, an experienced home brewery slightly better, and an experienced judge slightly better yet, and the judge in training worst of all. Give them time, they will be at the top.

[Phil_M]

I agree with Jim, and also have seen that sort of thing happen at work. They're more likely to overthink the issue, and subsequently doubt their initial response a bit more.

This data just seems reasonable to me. It makes sense that on average, BJCP judges have palates that are, well, average.

I'd hope this would encourage folks with the time to study to become BJCP judges, especially if they've ever felt that their palates aren't good enough.

[brulosopher]

If you assume that most or all BJCP judges in training are craft beer enthusiasts and home brewers, accounting for why they suddenly fall way short of their otherwise contemporaries could be explained by understanding how people learn a new skill. Frequently, while assimilating new knowledge, people drop dramatically in ability while they are struggling to get this new info to fit into their world.

I have taught a certain set of psychmotor skills to professionals in my day job. At the beginning of the session, if you have them perform an open skills test, using their existing skill level, they do fairly well. Then teach them several new skills for a few hours. Then go back to the original test, trying to use the new skills. They almost always perform worse than they did before the learning. Thats why its a futile plan to teach too much too soon when it comes to psychomotor skills. Analytically tasting beer is a psychomotor skill. Its hard for new judges to get out of their head. It makes perfect sense to me that a craft beer enthusiast would do fairly well, an experienced home brewery slightly better, and an experienced judge slightly better yet, and the judge in training worst of all. Give them time, they will be at the top.

Something to note-- many of the provisional judges from a year ago are now recognized or certified judges today, and the provisional judges today will likely be recognized or certified a year from now. I'm not convinced their performance was hindered by overthinking at all, but rather the fact any differences between the samples simply weren't noticeable. I'm biased, of course, as I taste a large majority of the xBmt beers and triangle myself every time, it's not as easy as people think... it's not as easy as I thought it would be.

[dmtaylor]

Interesting data point that wasn't touched on in the article: the % correct responses for "provisional" judges was a full 20% lower than everyone else.

This is very interesting.

Nice work on the xbmt -- it needed to be done.  Pretty clear results.

[HoosierBrew]

All in all, seems pretty reasonable. Thanks, Marshall !

[charles1968]

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

[brulosopher]

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

[AmandaK]

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

I did two tastings at NHC. Got 50% wrong. 

[brulosopher]

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

I did two tastings at NHC. Got 50% wrong. 

I prefer to look at it as you getting half right! Haha.

[AmandaK]

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

I did two tastings at NHC. Got 50% wrong. 

I prefer to look at it as you getting half right! Haha.

Where either of those significant? I think I remember no, which means I'm some measure better than average!

[brulosopher]

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

I did two tastings at NHC. Got 50% wrong. 

I prefer to look at it as you getting half right! Haha.

Where either of those significant? I think I remember no, which means I'm some measure better than average!

Not even close to significant.

[jeffy]

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

I did two tastings at NHC. Got 50% wrong. 

I prefer to look at it as you getting half right! Haha.

Where either of those significant? I think I remember no, which means I'm some measure better than average!

Not even close to significant.

Well, I got 'em both right and I couldn't smell anything.  Does that make me insignificant?