Author Topic: Investigating the Bad Palates Argument | A Graphical Look At xBmt Performance  (Read 2938 times)

Offline santoch

  • Brewmaster
  • *****
  • Posts: 887
  • North Bend, WA
    • View Profile
    • WAHA
This doesn't surprise me, either.  We all perceive thing different ways, and at different thresholds.
Mt. Si Brewing Society
Washington Homebrewer's Association (WAHA)
BJCP GM2/Mead Judge

Offline brulosopher

  • Brewmaster
  • *****
  • Posts: 505
  • They who drink beer will think beer
    • View Profile
    • Brülosophy


Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

I did two tastings at NHC. Got 50% wrong.  ;)

I prefer to look at it as you getting half right! Haha.
Where either of those significant? I think I remember no, which means I'm some measure better than average!

Not even close to significant.
Well, I got 'em both right and I couldn't smell anything.  Does that make me insignificant?

Of course not! Just statistically lucky  :-\

Offline narvin

  • Brewmaster General
  • *******
  • Posts: 2269
  • Baltimore
    • View Profile
In the BJCP, like anything else in life, status is a combination of skill and ambition.  We all know that there are bad judges out there.  Have you thought about going back and  collecting individual statistics, a "batting average" if you will, for all of your participants across every experiment?  You may find that some people are better tasters than others, even if it doesn't correlate to BJCP membership.
Please do not reply if your[sic] an evil alien!
Thanks

Offline charles1968

  • Brewmaster
  • *****
  • Posts: 533
    • View Profile

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

Interesting. I wouldn't expect judges to do any better than average, but I thought you might by luck have netted a supertaster or two among your volunteers or just a few people with better than average taste/smell sensitivity. Doesn't sound like it though. Supertasters tend to be Asian women - not so common in craft beer circles.

Offline hopfenundmalz

  • Global Moderator
  • I must live here
  • *****
  • Posts: 9005
  • Milford, MI
    • View Profile
In the BJCP, like anything else in life, status is a combination of skill and ambition.  We all know that there are bad judges out there.  Have you thought about going back and  collecting individual statistics, a "batting average" if you will, for all of your participants across every experiment?  You may find that some people are better tasters than others, even if it doesn't correlate to BJCP membership.
Add to that train of though - everyone has a bad day. Sometimes my allergies act up. And so on.

Then there are people that might be flavor blind to what the difference is.
Jeff Rankert
Ann Arbor Brewers Guild
AHA Governing Committee
BJCP National
Home-brewing, not just a hobby, it is a lifestyle!

Offline ynotbrusum

  • I spend way too much time on the AHA forum
  • ********
  • Posts: 2957
    • View Profile
I am quite certain that my training with a BJCP class improved my ability to detect certain aspects of beer that had previously flown under my radar, but I am also cognizant of the fact that my palate is in constant erosion in terms of sensitivity - so I really enjoy being paired with a younger palate to see what they get out of a particular beer.

As to statistical analysis of something so subjective as beer evaluation, I lean towards Mark Twain's comment: "Lies, Damn Lies, and Statistics."  Yet I really enjoy your XBmts, so please continue...we may yet crack the "Beer Genome".
Hodge Garage Brewing: "Brew with a glad heart!"

Offline brulosopher

  • Brewmaster
  • *****
  • Posts: 505
  • They who drink beer will think beer
    • View Profile
    • Brülosophy


Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

Interesting. I wouldn't expect judges to do any better than average, but I thought you might by luck have netted a supertaster or two among your volunteers or just a few people with better than average taste/smell sensitivity. Doesn't sound like it though. Supertasters tend to be Asian women - not so common in craft beer circles.

What's interesting to me is that 3 of my most reliable participants, guys who have missed maybe only 3-4 xBmts, are Certified judges and, in my opinion, great beer evaluators... 2 were a hair under 50% accurate and 1 was a hair over last I checked. My Coors Light drinking neighbor, on the other hand, a dude who vehemently hates IPA because it tastes "soapy" was closer to 60% accurate, and he has no clue what he's looking for.

In the BJCP, like anything else in life, status is a combination of skill and ambition.  We all know that there are bad judges out there.  Have you thought about going back and  collecting individual statistics, a "batting average" if you will, for all of your participants across every experiment?  You may find that some people are better tasters than others, even if it doesn't correlate to BJCP membership.
Add to that train of though - everyone has a bad day. Sometimes my allergies act up. And so on.

Then there are people that might be flavor blind to what the difference is.

Even if a person can't tell WHAT the difference is, we're interested simply in their ability to detect a difference.

I am quite certain that my training with a BJCP class improved my ability to detect certain aspects of beer that had previously flown under my radar, but I am also cognizant of the fact that my palate is in constant erosion in terms of sensitivity - so I really enjoy being paired with a younger palate to see what they get out of a particular beer.

As to statistical analysis of something so subjective as beer evaluation, I lean towards Mark Twain's comment: "Lies, Damn Lies, and Statistics."  Yet I really enjoy your XBmts, so please continue...we may yet crack the "Beer Genome".

I don't disagree, I just find so many things interesting!

Offline klickitat jim

  • I must live here
  • **********
  • Posts: 8604
    • View Profile

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

Interesting. I wouldn't expect judges to do any better than average, but I thought you might by luck have netted a supertaster or two among your volunteers or just a few people with better than average taste/smell sensitivity. Doesn't sound like it though. Supertasters tend to be Asian women - not so common in craft beer circles.
Asian females have best palates. Check
I am quite certain that my training with a BJCP class improved my ability to detect certain aspects of beer that had previously flown under my radar, but I am also cognizant of the fact that my palate is in constant erosion in terms of sensitivity - so I really enjoy being paired with a younger palate to see what they get out of a particular beer.

As to statistical analysis of something so subjective as beer evaluation, I lean towards Mark Twain's comment: "Lies, Damn Lies, and Statistics."  Yet I really enjoy your XBmts, so please continue...we may yet crack the "Beer Genome".
Team up with younger palate. Check


Offline denny

  • Administrator
  • Retired with too much time on my hands
  • *****
  • Posts: 19924
  • Noti OR [1991.4, 287.6deg] AR
    • View Profile
    • Dennybrew
Yep, I'm discovering that as I age my sense of smell, and therefore taste, comes and goes.  Couple that with a reduced tolerance of alcohol and you get the reasons I seldom judge any more.
Life begins at 60.....1.060, that is!

www.dennybrew.com

The best, sharpest, funniest, weirdest and most knowledgable minds in home brewing contribute on the AHA forum. - Alewyfe

"The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts." - Bertrand Russell

Offline mchrispen

  • Brewmaster
  • *****
  • Posts: 657
    • View Profile
    • Accidentalis Brewing Blog
It just occurred to me (I am a bit slow) that while the data analysis is interesting, it is correlated from tests that are NOT testing the test taker - rather testing for a specific difference in beer. I may misunderstand how triangles really work, but the foci is on the test. We might be able to correlate tester confidence in their answers, but not measure their specific skills by reverse engineering this data.


In other words, when people say, I am good at taking triangle tests, that is a meaningless statement. Specifically that being right or wrong is null in the analysis and a triangle test levels the playing field between unskilled and skilled when applied in this manner. It is a form of implied confirmation bias that there SHOULD be a difference between the beers, when statistically there may NOT be a difference, based solely on the tasting skills.


I feel like we always get caught up in our hypothesis that we should taste the difference, and that somehow our skills are lesser if we do not.


Then again, I might just suck at triangle taste testing :)
Matt Chrispen
Sometime Austin Zealot
Blogging from the garage @ accidentalis.com
>> Bru'n Water Spreadsheet Walkthroughs<<
>> Bru'n Water Subscriber Version 5.3 Spreadsheet Walkthrough <<

Offline klickitat jim

  • I must live here
  • **********
  • Posts: 8604
    • View Profile
Good point Matt. In taste analysis of beer, lack of confidence probably means second guessing. Something people who are in the process of learning new skills tend to do. In other words, I still believe the stat of judges in training being the worst at accuracy in blind triangle testing makes perfect sense.

Offline udubdawg

  • Senior Brewmaster
  • ******
  • Posts: 1080
    • View Profile
just going to toss this out there and leave the room   ;D

"BJCP-Trained and Sh**ty Palates are not mutually exclusive"

thanks for the data.


Offline charles1968

  • Brewmaster
  • *****
  • Posts: 533
    • View Profile
My Coors Light drinking neighbor, on the other hand, a dude who vehemently hates IPA because it tastes "soapy" was closer to 60% accurate, and he has no clue what he's looking for.

Could be interesting if he maintains that performance - he might be your star taster.

Offline charles1968

  • Brewmaster
  • *****
  • Posts: 533
    • View Profile

Offline charles1968

  • Brewmaster
  • *****
  • Posts: 533
    • View Profile
It just occurred to me (I am a bit slow) that while the data analysis is interesting, it is correlated from tests that are NOT testing the test taker - rather testing for a specific difference in beer. I may misunderstand how triangles really work, but the foci is on the test. We might be able to correlate tester confidence in their answers, but not measure their specific skills by reverse engineering this data.


In other words, when people say, I am good at taking triangle tests, that is a meaningless statement. Specifically that being right or wrong is null in the analysis and a triangle test levels the playing field between unskilled and skilled when applied in this manner. It is a form of implied confirmation bias that there SHOULD be a difference between the beers, when statistically there may NOT be a difference, based solely on the tasting skills.


I feel like we always get caught up in our hypothesis that we should taste the difference, and that somehow our skills are lesser if we do not.


Then again, I might just suck at triangle taste testing :)

Yes, it's likely there really is no perceptible difference caused by a lot of the variables tested. However, a second possibility is that there is a difference but it's under most people's taste threshold (and therefore doesn't matter). A third possibility is that something about triangle tests (palate fatigue / sipping small amounts / mixing successive drinks in the mouth and nasal cavity) makes triangle tasters less sensitive than regular beer drinkers.

I lean towards option 1, if only because I enjoy the fact it undermines 95% of received wisdom about how to brew.
« Last Edit: January 22, 2016, 06:48:32 PM by charles1968 »