Homebrewers Association | AHA Forum

General Category => General Homebrew Discussion => Topic started by: brulosopher on January 21, 2016, 01:03:24 PM

Title: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: brulosopher on January 21, 2016, 01:03:24 PM
The rate of statistically significant exBEERiment findings currently hovers around 35%, a respectable number in my opinion, though far lower than many have expected given the presumable importance of the variables investigated. While it's only natural to come up with excuses as to how this could be, none have been more popular than what I've come to call... The s***ty Palates Argument. Greg and I sought the assistance of bloggers Scott Janish and Justin Angevaare to help us compare triangle test performance rates based on level of experience. Are BJCP judges actually better at distinguishing differences than general beer drinkers? Read to find out!

http://brulosophy.com/2016/01/21/investigating-the-bad-palates-argument-a-graphical-look-at-xbmt-performance-based-on-experience-level/
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: AmandaK on January 21, 2016, 01:19:43 PM
The rate of statistically significant exBEERiment findings currently hovers around 35%, a respectable number in my opinion, though far lower than many have expected given the presumable importance of the variables investigated. While it's only natural to come up with excuses as to how this could be, none have been more popular than what I've come to call... The s***ty Palates Argument. Greg and I sought the assistance of bloggers Scott Janish and Justin Angevaare to help us compare triangle test performance rates based on level of experience. Are BJCP judges actually better at distinguishing differences than general beer drinkers? Read to find out!

http://brulosophy.com/2016/01/21/investigating-the-bad-palates-argument-a-graphical-look-at-xbmt-performance-based-on-experience-level/

Side bar: I love that you got Janish in on this.

Interesting data point that wasn't touched on in the article: the % correct responses for "provisional" judges was a full 20% lower than everyone else. Any thoughts on why that is? Lower sample size or something more intriguing?
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: brulosopher on January 21, 2016, 01:23:25 PM
The rate of statistically significant exBEERiment findings currently hovers around 35%, a respectable number in my opinion, though far lower than many have expected given the presumable importance of the variables investigated. While it's only natural to come up with excuses as to how this could be, none have been more popular than what I've come to call... The s***ty Palates Argument. Greg and I sought the assistance of bloggers Scott Janish and Justin Angevaare to help us compare triangle test performance rates based on level of experience. Are BJCP judges actually better at distinguishing differences than general beer drinkers? Read to find out!

http://brulosophy.com/2016/01/21/investigating-the-bad-palates-argument-a-graphical-look-at-xbmt-performance-based-on-experience-level/

Side bar: I love that you got Janish in on this.

Interesting data point that wasn't touched on in the article: the % correct responses for "provisional" judges was a full 20% lower than everyone else. Any thoughts on why that is? Lower sample size or something more intriguing?

Scott is a badass!

I noticed that odd stat as well and honestly have no clue why it occurred. What i find most curious is the fact what you pointed out is only true when looking at only significant xBmts, as provisional judges performed about the same as recognized and higher on all xBmts combined. Anomaly perhaps?
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: klickitat jim on January 21, 2016, 02:07:58 PM
If you assume that most or all BJCP judges in training are craft beer enthusiasts and home brewers, accounting for why they suddenly fall way short of their otherwise contemporaries could be explained by understanding how people learn a new skill. Frequently, while assimilating new knowledge, people drop dramatically in ability while they are struggling to get this new info to fit into their world.

I have taught a certain set of psychmotor skills to professionals in my day job. At the beginning of the session, if you have them perform an open skills test, using their existing skill level, they do fairly well. Then teach them several new skills for a few hours. Then go back to the original test, trying to use the new skills. They almost always perform worse than they did before the learning. Thats why its a futile plan to teach too much too soon when it comes to psychomotor skills. Analytically tasting beer is a psychomotor skill. Its hard for new judges to get out of their head. It makes perfect sense to me that a craft beer enthusiast would do fairly well, an experienced home brewery slightly better, and an experienced judge slightly better yet, and the judge in training worst of all. Give them time, they will be at the top.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: Phil_M on January 21, 2016, 02:25:55 PM
I agree with Jim, and also have seen that sort of thing happen at work. They're more likely to overthink the issue, and subsequently doubt their initial response a bit more.

This data just seems reasonable to me. It makes sense that on average, BJCP judges have palates that are, well, average.

I'd hope this would encourage folks with the time to study to become BJCP judges, especially if they've ever felt that their palates aren't good enough.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: brulosopher on January 21, 2016, 02:33:12 PM
If you assume that most or all BJCP judges in training are craft beer enthusiasts and home brewers, accounting for why they suddenly fall way short of their otherwise contemporaries could be explained by understanding how people learn a new skill. Frequently, while assimilating new knowledge, people drop dramatically in ability while they are struggling to get this new info to fit into their world.

I have taught a certain set of psychmotor skills to professionals in my day job. At the beginning of the session, if you have them perform an open skills test, using their existing skill level, they do fairly well. Then teach them several new skills for a few hours. Then go back to the original test, trying to use the new skills. They almost always perform worse than they did before the learning. Thats why its a futile plan to teach too much too soon when it comes to psychomotor skills. Analytically tasting beer is a psychomotor skill. Its hard for new judges to get out of their head. It makes perfect sense to me that a craft beer enthusiast would do fairly well, an experienced home brewery slightly better, and an experienced judge slightly better yet, and the judge in training worst of all. Give them time, they will be at the top.

Something to note-- many of the provisional judges from a year ago are now recognized or certified judges today, and the provisional judges today will likely be recognized or certified a year from now. I'm not convinced their performance was hindered by overthinking at all, but rather the fact any differences between the samples simply weren't noticeable. I'm biased, of course, as I taste a large majority of the xBmt beers and triangle myself every time, it's not as easy as people think... it's not as easy as I thought it would be.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: dmtaylor on January 21, 2016, 03:00:02 PM
Interesting data point that wasn't touched on in the article: the % correct responses for "provisional" judges was a full 20% lower than everyone else.

This is very interesting.

Nice work on the xbmt -- it needed to be done.  Pretty clear results.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: HoosierBrew on January 21, 2016, 03:03:13 PM
All in all, seems pretty reasonable. Thanks, Marshall !
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: charles1968 on January 21, 2016, 07:36:07 PM
Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: brulosopher on January 21, 2016, 11:54:15 PM

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: AmandaK on January 22, 2016, 12:19:12 AM

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

I did two tastings at NHC. Got 50% wrong.  ;)
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: brulosopher on January 22, 2016, 12:22:09 AM

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

I did two tastings at NHC. Got 50% wrong.  ;)

I prefer to look at it as you getting half right! Haha.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: AmandaK on January 22, 2016, 12:24:45 AM

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

I did two tastings at NHC. Got 50% wrong.  ;)

I prefer to look at it as you getting half right! Haha.
Where either of those significant? I think I remember no, which means I'm some measure better than average!
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: brulosopher on January 22, 2016, 12:25:31 AM


Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

I did two tastings at NHC. Got 50% wrong.  ;)

I prefer to look at it as you getting half right! Haha.
Where either of those significant? I think I remember no, which means I'm some measure better than average!

Not even close to significant.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: jeffy on January 22, 2016, 12:44:59 AM


Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

I did two tastings at NHC. Got 50% wrong.  ;)

I prefer to look at it as you getting half right! Haha.
Where either of those significant? I think I remember no, which means I'm some measure better than average!

Not even close to significant.
Well, I got 'em both right and I couldn't smell anything.  Does that make me insignificant?
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: santoch on January 22, 2016, 02:43:03 AM
This doesn't surprise me, either.  We all perceive thing different ways, and at different thresholds.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: brulosopher on January 22, 2016, 03:19:22 AM


Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

I did two tastings at NHC. Got 50% wrong.  ;)

I prefer to look at it as you getting half right! Haha.
Where either of those significant? I think I remember no, which means I'm some measure better than average!

Not even close to significant.
Well, I got 'em both right and I couldn't smell anything.  Does that make me insignificant?

Of course not! Just statistically lucky  :-\
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: narvin on January 22, 2016, 03:23:41 AM
In the BJCP, like anything else in life, status is a combination of skill and ambition.  We all know that there are bad judges out there.  Have you thought about going back and  collecting individual statistics, a "batting average" if you will, for all of your participants across every experiment?  You may find that some people are better tasters than others, even if it doesn't correlate to BJCP membership.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: charles1968 on January 22, 2016, 08:45:16 AM

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

Interesting. I wouldn't expect judges to do any better than average, but I thought you might by luck have netted a supertaster or two among your volunteers or just a few people with better than average taste/smell sensitivity. Doesn't sound like it though. Supertasters tend to be Asian women - not so common in craft beer circles.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: hopfenundmalz on January 22, 2016, 12:34:25 PM
In the BJCP, like anything else in life, status is a combination of skill and ambition.  We all know that there are bad judges out there.  Have you thought about going back and  collecting individual statistics, a "batting average" if you will, for all of your participants across every experiment?  You may find that some people are better tasters than others, even if it doesn't correlate to BJCP membership.
Add to that train of though - everyone has a bad day. Sometimes my allergies act up. And so on.

Then there are people that might be flavor blind to what the difference is.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: ynotbrusum on January 22, 2016, 01:01:10 PM
I am quite certain that my training with a BJCP class improved my ability to detect certain aspects of beer that had previously flown under my radar, but I am also cognizant of the fact that my palate is in constant erosion in terms of sensitivity - so I really enjoy being paired with a younger palate to see what they get out of a particular beer.

As to statistical analysis of something so subjective as beer evaluation, I lean towards Mark Twain's comment: "Lies, Damn Lies, and Statistics."  Yet I really enjoy your XBmts, so please continue...we may yet crack the "Beer Genome".
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: brulosopher on January 22, 2016, 01:51:08 PM


Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

Interesting. I wouldn't expect judges to do any better than average, but I thought you might by luck have netted a supertaster or two among your volunteers or just a few people with better than average taste/smell sensitivity. Doesn't sound like it though. Supertasters tend to be Asian women - not so common in craft beer circles.

What's interesting to me is that 3 of my most reliable participants, guys who have missed maybe only 3-4 xBmts, are Certified judges and, in my opinion, great beer evaluators... 2 were a hair under 50% accurate and 1 was a hair over last I checked. My Coors Light drinking neighbor, on the other hand, a dude who vehemently hates IPA because it tastes "soapy" was closer to 60% accurate, and he has no clue what he's looking for.

In the BJCP, like anything else in life, status is a combination of skill and ambition.  We all know that there are bad judges out there.  Have you thought about going back and  collecting individual statistics, a "batting average" if you will, for all of your participants across every experiment?  You may find that some people are better tasters than others, even if it doesn't correlate to BJCP membership.
Add to that train of though - everyone has a bad day. Sometimes my allergies act up. And so on.

Then there are people that might be flavor blind to what the difference is.

Even if a person can't tell WHAT the difference is, we're interested simply in their ability to detect a difference.

I am quite certain that my training with a BJCP class improved my ability to detect certain aspects of beer that had previously flown under my radar, but I am also cognizant of the fact that my palate is in constant erosion in terms of sensitivity - so I really enjoy being paired with a younger palate to see what they get out of a particular beer.

As to statistical analysis of something so subjective as beer evaluation, I lean towards Mark Twain's comment: "Lies, Damn Lies, and Statistics."  Yet I really enjoy your XBmts, so please continue...we may yet crack the "Beer Genome".

I don't disagree, I just find so many things interesting!
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: klickitat jim on January 22, 2016, 03:54:42 PM

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

Interesting. I wouldn't expect judges to do any better than average, but I thought you might by luck have netted a supertaster or two among your volunteers or just a few people with better than average taste/smell sensitivity. Doesn't sound like it though. Supertasters tend to be Asian women - not so common in craft beer circles.
Asian females have best palates. Check
I am quite certain that my training with a BJCP class improved my ability to detect certain aspects of beer that had previously flown under my radar, but I am also cognizant of the fact that my palate is in constant erosion in terms of sensitivity - so I really enjoy being paired with a younger palate to see what they get out of a particular beer.

As to statistical analysis of something so subjective as beer evaluation, I lean towards Mark Twain's comment: "Lies, Damn Lies, and Statistics."  Yet I really enjoy your XBmts, so please continue...we may yet crack the "Beer Genome".
Team up with younger palate. Check

Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: denny on January 22, 2016, 04:33:40 PM
Yep, I'm discovering that as I age my sense of smell, and therefore taste, comes and goes.  Couple that with a reduced tolerance of alcohol and you get the reasons I seldom judge any more.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: mchrispen on January 22, 2016, 04:43:00 PM
It just occurred to me (I am a bit slow) that while the data analysis is interesting, it is correlated from tests that are NOT testing the test taker - rather testing for a specific difference in beer. I may misunderstand how triangles really work, but the foci is on the test. We might be able to correlate tester confidence in their answers, but not measure their specific skills by reverse engineering this data.


In other words, when people say, I am good at taking triangle tests, that is a meaningless statement. Specifically that being right or wrong is null in the analysis and a triangle test levels the playing field between unskilled and skilled when applied in this manner. It is a form of implied confirmation bias that there SHOULD be a difference between the beers, when statistically there may NOT be a difference, based solely on the tasting skills.


I feel like we always get caught up in our hypothesis that we should taste the difference, and that somehow our skills are lesser if we do not.


Then again, I might just suck at triangle taste testing :)
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: klickitat jim on January 22, 2016, 04:50:59 PM
Good point Matt. In taste analysis of beer, lack of confidence probably means second guessing. Something people who are in the process of learning new skills tend to do. In other words, I still believe the stat of judges in training being the worst at accuracy in blind triangle testing makes perfect sense.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: udubdawg on January 22, 2016, 05:20:43 PM
just going to toss this out there and leave the room   ;D

"BJCP-Trained and Sh**ty Palates are not mutually exclusive"

thanks for the data.

Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: charles1968 on January 22, 2016, 06:38:58 PM
My Coors Light drinking neighbor, on the other hand, a dude who vehemently hates IPA because it tastes "soapy" was closer to 60% accurate, and he has no clue what he's looking for.

Could be interesting if he maintains that performance - he might be your star taster.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: charles1968 on January 22, 2016, 06:40:29 PM
Asian females have best palates. Check

LOL
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: charles1968 on January 22, 2016, 06:47:01 PM
It just occurred to me (I am a bit slow) that while the data analysis is interesting, it is correlated from tests that are NOT testing the test taker - rather testing for a specific difference in beer. I may misunderstand how triangles really work, but the foci is on the test. We might be able to correlate tester confidence in their answers, but not measure their specific skills by reverse engineering this data.


In other words, when people say, I am good at taking triangle tests, that is a meaningless statement. Specifically that being right or wrong is null in the analysis and a triangle test levels the playing field between unskilled and skilled when applied in this manner. It is a form of implied confirmation bias that there SHOULD be a difference between the beers, when statistically there may NOT be a difference, based solely on the tasting skills.


I feel like we always get caught up in our hypothesis that we should taste the difference, and that somehow our skills are lesser if we do not.


Then again, I might just suck at triangle taste testing :)

Yes, it's likely there really is no perceptible difference caused by a lot of the variables tested. However, a second possibility is that there is a difference but it's under most people's taste threshold (and therefore doesn't matter). A third possibility is that something about triangle tests (palate fatigue / sipping small amounts / mixing successive drinks in the mouth and nasal cavity) makes triangle tasters less sensitive than regular beer drinkers.

I lean towards option 1, if only because I enjoy the fact it undermines 95% of received wisdom about how to brew.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: Phil_M on January 22, 2016, 07:12:48 PM
My Coors Light drinking neighbor, on the other hand, a dude who vehemently hates IPA because it tastes "soapy" was closer to 60% accurate, and he has no clue what he's looking for.

Could be interesting if he maintains that performance - he might be your star taster.

I know some folks who are huge in to Budweiser, as in back in the day they'd travel to the different breweries and compare the product. I wonder how sensitive their palates are, as they were able to distinguish minute differences in a pretty flavorless beer.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: Frankenbrew on January 23, 2016, 06:00:27 PM
My Coors Light drinking neighbor, on the other hand, a dude who vehemently hates IPA because it tastes "soapy" was closer to 60% accurate, and he has no clue what he's looking for.

Could be interesting if he maintains that performance - he might be your star taster.

I know some folks who are huge in to Budweiser, as in back in the day they'd travel to the different breweries and compare the product. I wonder how sensitive their palates are, as they were able to distinguish minute differences in a pretty flavorless beer.

I find this hilarious!
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: brulosopher on January 24, 2016, 01:42:30 AM

My Coors Light drinking neighbor, on the other hand, a dude who vehemently hates IPA because it tastes "soapy" was closer to 60% accurate, and he has no clue what he's looking for.

Could be interesting if he maintains that performance - he might be your star taster.

I know some folks who are huge in to Budweiser, as in back in the day they'd travel to the different breweries and compare the product. I wonder how sensitive their palates are, as they were able to distinguish minute differences in a pretty flavorless beer.

I can reliably distinguish Coors Light, Bud Light, and Miller Light. No sh!t.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: majorvices on January 24, 2016, 02:29:35 AM
I just gotta say, this was my favorite xBmt experiment thus far. Very good stuff!
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: neddles on January 24, 2016, 05:29:23 AM
I just gotta say, this was my favorite xBmt experiment thus far. Very good stuff!
Was just like the other xBmts as far as I could tell. :)
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: majorvices on January 24, 2016, 11:59:48 AM
I just gotta say, this was my favorite xBmt experiment thus far. Very good stuff!
Was just like the other xBmts as far as I could tell. :)

I like all his experiments but this one is something along the lines I have been thinking for a very long time.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: neddles on January 24, 2016, 01:50:18 PM
I just gotta say, this was my favorite xBmt experiment thus far. Very good stuff!
Was just like the other xBmts as far as I could tell. :)

I like all his experiments but this one is something along the lines I have been thinking for a very long time.
I completely agree with you here. My prior comment was apparently a poor joke.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: Hooper on January 26, 2016, 02:18:21 AM
Results:

1)  Follow good brewing practices = good beer

2)  Enter a brewing contest = crap shoot
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: mabrungard on March 09, 2017, 03:40:50 PM
Since this is allied with this subject, I'm carrying on here.

I do find that untrained palates are often less likely to discern differences in beer flavor and character. When tasting panels are convened in commercial settings, they are often trained and graded for sensitivity in a number of sensory areas. I'm concerned that the use of 'regular' untrained tasters that have no guidance as to what differences they should be looking for or an ability to recognize them, leaves this testing result with an overly skewed result of...can't tell a difference. Its not until you have a 'clubbed over the head' difference in beers that a viable result can be noted. I feel that's not good for science and not good for brewing improvement.

While I applaud the explorations that Brulosophy conducts, the results point out the mediocrity of an untrained palate that has no idea of what it might need to note as a difference. Since most of these tests compare nuanced differences, it is probably also appropriate to include more focused assessments and comparisons using trained palates to help discern if there are differences. I like that the authors of these various exbeeriments often try to explore differences in their beers with their full knowledge of their brewing differences, but I'd like to see more trained palates included in that assessment. Triangle testing does help reduce randomness in the assessments, but I would like to know that there has been an opportunity for the taster to focus on what the potential difference or flaw is and if its really perceptible.

Since these beers are often decent, similar beers, I'm not surprised that the tasters can't perceive a difference between them. But I don't want to automatically apply a finding of 'makes no statistical difference' to an experimental trial with that measurement alone. Remember, the majority of beer drinkers think that Budmilloors is great beer. 
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: bayareabrewer on March 09, 2017, 04:21:49 PM

My Coors Light drinking neighbor, on the other hand, a dude who vehemently hates IPA because it tastes "soapy" was closer to 60% accurate, and he has no clue what he's looking for.

Could be interesting if he maintains that performance - he might be your star taster.

I know some folks who are huge in to Budweiser, as in back in the day they'd travel to the different breweries and compare the product. I wonder how sensitive their palates are, as they were able to distinguish minute differences in a pretty flavorless beer.

I can reliably distinguish Coors Light, Bud Light, and Miller Light. No sh!t.

Where are your tastings taking place? What setting?
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: denny on March 09, 2017, 05:03:50 PM
Since this is allied with this subject, I'm carrying on here.

I do find that untrained palates are often less likely to discern differences in beer flavor and character. When tasting panels are convened in commercial settings, they are often trained and graded for sensitivity in a number of sensory areas. I'm concerned that the use of 'regular' untrained tasters that have no guidance as to what differences they should be looking for or an ability to recognize them, leaves this testing result with an overly skewed result of...can't tell a difference. Its not until you have a 'clubbed over the head' difference in beers that a viable result can be noted. I feel that's not good for science and not good for brewing improvement.

While I applaud the explorations that Brulosophy conducts, the results point out the mediocrity of an untrained palate that has no idea of what it might need to note as a difference. Since most of these tests compare nuanced differences, it is probably also appropriate to include more focused assessments and comparisons using trained palates to help discern if there are differences. I like that the authors of these various exbeeriments often try to explore differences in their beers with their full knowledge of their brewing differences, but I'd like to see more trained palates included in that assessment. Triangle testing does help reduce randomness in the assessments, but I would like to know that there has been an opportunity for the taster to focus on what the potential difference or flaw is and if its really perceptible.

Since these beers are often decent, similar beers, I'm not surprised that the tasters can't perceive a difference between them. But I don't want to automatically apply a finding of 'makes no statistical difference' to an experimental trial with that measurement alone. Remember, the majority of beer drinkers think that Budmilloors is great beer.

Martin, keep in mind that in general these experiments are not like judging a comp.  I know that I, and I think Drew and the Brulosophy guys, are more concerned with what a "normal" beer drinker perceives and their overall enjoyment of the beer.  I think what you're talking about is different than the purposes we all have in mind.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: mabrungard on March 09, 2017, 10:12:25 PM
Martin, keep in mind that in general these experiments are not like judging a comp.  I know that I, and I think Drew and the Brulosophy guys, are more concerned with what a "normal" beer drinker perceives and their overall enjoyment of the beer. 

I agree with you, but my problem with these results is that people are considering them as gospel and this form of testing amounts to a single data point that may not say anything definitive. Yet brewers are considering that single data point definitive...almost like alternative facts from the internet.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: denny on March 09, 2017, 10:15:03 PM
Martin, keep in mind that in general these experiments are not like judging a comp.  I know that I, and I think Drew and the Brulosophy guys, are more concerned with what a "normal" beer drinker perceives and their overall enjoyment of the beer. 

I agree with you, but my problem with these results is that people are considering them as gospel and this form of testing amounts to a single data point that may not say anything definitive. Yet brewers are considering that single data point definitive...almost like alternative facts from the internet.

Unfortunately, we have no control over how people take the experiments, no matter how often we say that it's just a data point.  That's not our fault.  But it will be one of the things that the 4 of us talk about in our seminar in June.
Title: Re: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance
Post by: Phil_M on March 10, 2017, 01:01:18 AM
Unfortunately, we have no control over how people take the experiments, no matter how often we say that it's just a data point.  That's not our fault.  But it will be one of the things that the 4 of us talk about in our seminar in June.

Bingo. "Don't shoot the messanger."

True, how things are presented matters, but not everyone is going to take away the message you attempted to convey. You could waste a lot of energy trying to be perfectly "politically" correct in this regard.