[santoch]

This doesn't surprise me, either.  We all perceive thing different ways, and at different thresholds.

[brulosopher]

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

I did two tastings at NHC. Got 50% wrong. 

I prefer to look at it as you getting half right! Haha.

Where either of those significant? I think I remember no, which means I'm some measure better than average!

Not even close to significant.

Well, I got 'em both right and I couldn't smell anything.  Does that make me insignificant?

Of course not! Just statistically lucky 

[narvin]

In the BJCP, like anything else in life, status is a combination of skill and ambition.  We all know that there are bad judges out there.  Have you thought about going back and  collecting individual statistics, a "batting average" if you will, for all of your participants across every experiment?  You may find that some people are better tasters than others, even if it doesn't correlate to BJCP membership.

[charles1968]

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

Interesting. I wouldn't expect judges to do any better than average, but I thought you might by luck have netted a supertaster or two among your volunteers or just a few people with better than average taste/smell sensitivity. Doesn't sound like it though. Supertasters tend to be Asian women - not so common in craft beer circles.

[hopfenundmalz]

In the BJCP, like anything else in life, status is a combination of skill and ambition.  We all know that there are bad judges out there.  Have you thought about going back and  collecting individual statistics, a "batting average" if you will, for all of your participants across every experiment?  You may find that some people are better tasters than others, even if it doesn't correlate to BJCP membership.

Add to that train of though - everyone has a bad day. Sometimes my allergies act up. And so on.

Then there are people that might be flavor blind to what the difference is.

[ynotbrusum]

I am quite certain that my training with a BJCP class improved my ability to detect certain aspects of beer that had previously flown under my radar, but I am also cognizant of the fact that my palate is in constant erosion in terms of sensitivity - so I really enjoy being paired with a younger palate to see what they get out of a particular beer.

As to statistical analysis of something so subjective as beer evaluation, I lean towards Mark Twain's comment: "Lies, Damn Lies, and Statistics."  Yet I really enjoy your XBmts, so please continue...we may yet crack the "Beer Genome".

[brulosopher]

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

Interesting. I wouldn't expect judges to do any better than average, but I thought you might by luck have netted a supertaster or two among your volunteers or just a few people with better than average taste/smell sensitivity. Doesn't sound like it though. Supertasters tend to be Asian women - not so common in craft beer circles.

What's interesting to me is that 3 of my most reliable participants, guys who have missed maybe only 3-4 xBmts, are Certified judges and, in my opinion, great beer evaluators... 2 were a hair under 50% accurate and 1 was a hair over last I checked. My Coors Light drinking neighbor, on the other hand, a dude who vehemently hates IPA because it tastes "soapy" was closer to 60% accurate, and he has no clue what he's looking for.

In the BJCP, like anything else in life, status is a combination of skill and ambition.  We all know that there are bad judges out there.  Have you thought about going back and  collecting individual statistics, a "batting average" if you will, for all of your participants across every experiment?  You may find that some people are better tasters than others, even if it doesn't correlate to BJCP membership.

Add to that train of though - everyone has a bad day. Sometimes my allergies act up. And so on.

Then there are people that might be flavor blind to what the difference is.

Even if a person can't tell WHAT the difference is, we're interested simply in their ability to detect a difference.

I am quite certain that my training with a BJCP class improved my ability to detect certain aspects of beer that had previously flown under my radar, but I am also cognizant of the fact that my palate is in constant erosion in terms of sensitivity - so I really enjoy being paired with a younger palate to see what they get out of a particular beer.

As to statistical analysis of something so subjective as beer evaluation, I lean towards Mark Twain's comment: "Lies, Damn Lies, and Statistics."  Yet I really enjoy your XBmts, so please continue...we may yet crack the "Beer Genome".

I don't disagree, I just find so many things interesting!

[klickitat jim]

Thumbs up.

Is there any evidence in the data that specific individuals consistently do better than average (supertasters)? If so, shortlisting them might make taste tests more sensitive (though possibly to things that don't matter...)

We can look at specific individuals this time around, but I have in the past, about five months ago I believe, and there was no pattern whatsoever-- even higher ranked judges were wrong about 50% of the time.

Interesting. I wouldn't expect judges to do any better than average, but I thought you might by luck have netted a supertaster or two among your volunteers or just a few people with better than average taste/smell sensitivity. Doesn't sound like it though. Supertasters tend to be Asian women - not so common in craft beer circles.

Asian females have best palates. Check

I am quite certain that my training with a BJCP class improved my ability to detect certain aspects of beer that had previously flown under my radar, but I am also cognizant of the fact that my palate is in constant erosion in terms of sensitivity - so I really enjoy being paired with a younger palate to see what they get out of a particular beer.

As to statistical analysis of something so subjective as beer evaluation, I lean towards Mark Twain's comment: "Lies, Damn Lies, and Statistics."  Yet I really enjoy your XBmts, so please continue...we may yet crack the "Beer Genome".

Team up with younger palate. Check

[denny]

Yep, I'm discovering that as I age my sense of smell, and therefore taste, comes and goes.  Couple that with a reduced tolerance of alcohol and you get the reasons I seldom judge any more.

[mchrispen]

It just occurred to me (I am a bit slow) that while the data analysis is interesting, it is correlated from tests that are NOT testing the test taker - rather testing for a specific difference in beer. I may misunderstand how triangles really work, but the foci is on the test. We might be able to correlate tester confidence in their answers, but not measure their specific skills by reverse engineering this data.


In other words, when people say, I am good at taking triangle tests, that is a meaningless statement. Specifically that being right or wrong is null in the analysis and a triangle test levels the playing field between unskilled and skilled when applied in this manner. It is a form of implied confirmation bias that there SHOULD be a difference between the beers, when statistically there may NOT be a difference, based solely on the tasting skills.


I feel like we always get caught up in our hypothesis that we should taste the difference, and that somehow our skills are lesser if we do not.


Then again, I might just suck at triangle taste testing

[klickitat jim]

Good point Matt. In taste analysis of beer, lack of confidence probably means second guessing. Something people who are in the process of learning new skills tend to do. In other words, I still believe the stat of judges in training being the worst at accuracy in blind triangle testing makes perfect sense.

[udubdawg]

just going to toss this out there and leave the room   

"BJCP-Trained and Sh**ty Palates are not mutually exclusive"

thanks for the data.

[charles1968]

My Coors Light drinking neighbor, on the other hand, a dude who vehemently hates IPA because it tastes "soapy" was closer to 60% accurate, and he has no clue what he's looking for.

Could be interesting if he maintains that performance - he might be your star taster.

[charles1968]

Asian females have best palates. Check

LOL

[charles1968]

It just occurred to me (I am a bit slow) that while the data analysis is interesting, it is correlated from tests that are NOT testing the test taker - rather testing for a specific difference in beer. I may misunderstand how triangles really work, but the foci is on the test. We might be able to correlate tester confidence in their answers, but not measure their specific skills by reverse engineering this data.


In other words, when people say, I am good at taking triangle tests, that is a meaningless statement. Specifically that being right or wrong is null in the analysis and a triangle test levels the playing field between unskilled and skilled when applied in this manner. It is a form of implied confirmation bias that there SHOULD be a difference between the beers, when statistically there may NOT be a difference, based solely on the tasting skills.


I feel like we always get caught up in our hypothesis that we should taste the difference, and that somehow our skills are lesser if we do not.


Then again, I might just suck at triangle taste testing

Yes, it's likely there really is no perceptible difference caused by a lot of the variables tested. However, a second possibility is that there is a difference but it's under most people's taste threshold (and therefore doesn't matter). A third possibility is that something about triangle tests (palate fatigue / sipping small amounts / mixing successive drinks in the mouth and nasal cavity) makes triangle tasters less sensitive than regular beer drinkers.

I lean towards option 1, if only because I enjoy the fact it undermines 95% of received wisdom about how to brew.