[klickitat jim]

I have a different view, more plain. To me barely beating the odds isn't "significant" though statistically it might be. Using your 2/5 one in ten odds of randomly correct, if you had ten tasters and one chose correctly, it might be random. It also might not be. If two people chose correctly, maybe one is random and the other correct. There's 8 more who got it wrong though, so to me it's still not a significant difference. Significant statist of correct choosers maybe. In my simple mind, if all ten chose right, that's probably a big difference. Whatever, what I'm driving at is that a statistician says they found a statistically significant difference, and that gets translated into significant difference in layman's terms.

[Andor]

As an outsider who has never done a triangle test or had a beer judged, I wonder how much the carry over of flavors between similar beers skews the results when ever I read the results of the brulosophy stuff. An extreme example would be a chilli triangle test, two sample with ghost peppers one with jalapeño  If you know ghost peppers they are no jalapeño but I'm sure if some got the ghost as sample one and two they would have a hard time with sample 3. Extreme example but I wonder how much this comes into play with  strong flavored beers as well- IPA, stout etc

[erockrph]

I have a different view, more plain. To me barely beating the odds isn't "significant" though statistically it might be. Using your 2/5 one in ten odds of randomly correct, if you had ten tasters and one chose correctly, it might be random. It also might not be. If two people chose correctly, maybe one is random and the other correct. There's 8 more who got it wrong though, so to me it's still not a significant difference. Significant statist of correct choosers maybe. In my simple mind, if all ten chose right, that's probably a big difference. Whatever, what I'm driving at is that a statistician says they found a statistically significant difference, and that gets translated into significant difference in layman's terms.

FYI, that's what the p-value checks for. When a Brulosophy experiment returns a significant result, then there is at least a 95% chance that it is not due to random guessing. The experiment results need to return enough positive results beyond what you would expect from a random guess, so that you can be 95% sure there is a positive correlation.

And this is why results that don't reach the statistical significance mark have to be taken with a grain of salt. An experiment that may hit the 85% or 90% mark would not be considered statistically significant based on the p-value that Brulosophy has chosen. That doesn't mean that there's not a correlation. It just means that the results haven't hit the threshold where they're 95% confident that it's not random chance.

[klickitat jim]

I think I get that. But what I'm saying is there is a difference between "statistically significant results" and a plain old "significant difference ".

There may be 100% of tasters accurately choose a bud from a bud light, statistically significant results... but that doesn't mean that bud is significantly different than bud light. If it does mean there's a significant difference then what level of difference would we call a bud compared to a Bigfoot?

[Phil_M]

If it does mean there's a significant difference then what level of difference would we call a bud compared to a Bigfoot?

Depends on if I've just finished cutting grass, or if I'm sitting in front of the fire with a book.

[klickitat jim]

Makes sense. Statistically significant sense.

[dbeechum]

There may be 100% of tasters accurately choose a bud from a bud light, statistically significant results.

And what's great - if you run the experiment - testers get it right about 33% of the time. e.g. random chance.

[Andy Farke]

This is a fascinating thread, which coincidentally I found only after I posted this (provacatively titled) blog entry last night ("Are Homebrew Experiments Scientific?"): https://andybrews.com/2017/02/07/are-homebrew-experiments-scientific/

The tl;dr on it is "Yes, and No," for a variety of reasons (many of them eloquently outlined previously by Brülosophy and Experimental Brewing posts, too). In many cases (in my opinion), the issue is not so much with the experiments or the experimenters--it is how outside readers interpret the experiments, as echoed by others in this thread.

Elsewhere in this thread, I saw something along the lines of "Well, it's only homebrewing, and not done in a lab, so it can't be that scientific." (my paraphrase and some creative interpretation, not a direct quote) As a working scientist, I can say that is a potentially incorrect belief. An experiment is only as scientific as its protocol and interpretation, regardless of whether it is done with fancy equipment or in an expensive, shiny room. Most home setups are limited in the kinds of tests they can do (I don't know any homebrewer with a mass spec in the garage, for instance), but hey, that's why you send stuff out for analysis. If a homebrewer is careful, sets up testable hypotheses, documents their process in detail, interprets the results cautiously, and revises their interpretations in response to constructive review, they are just as scientific as a fancy lab (and quite frankly, sometimes more scientific).

[denny]

Well said, Andy!  At Homebrew on this summer, Drew, Marshall, Malcolm and I will be doing a seminar together where we'll talk about how we set up and interpret experiments in an effort to help people understand better what we do and how they can use the info.

[Andy Farke]

Well said, Andy!  At Homebrew on this summer, Drew, Marshall, Malcolm and I will be doing a seminar together where we'll talk about how we set up and interpret experiments in an effort to help people understand better what we do and how they can use the info.

Thanks!

And the seminar sounds awesome! I'll be there (and will be giving a seminar too, on the connections between paleontology, geology, and homebrewing), and hope we can chat about experimental brewing then.

One thing that seems to be a really common area of misconception and misinterpretation is the P value. Hmm...might have to write up a post on that, too!

[dmtaylor]

One thing that seems to be a really common area of misconception and misinterpretation is the P value. Hmm...might have to write up a post on that, too!

Please do.  Knowledge is power!  And thank you for your input thus far.  Cheers.

[denny]

One thing that seems to be a really common area of misconception and misinterpretation is the P value. Hmm...might have to write up a post on that, too!

Please do.  Knowledge is power!  And thank you for your input thus far.  Cheers.

http://simplystatistics.org/2015/08/19/p-0-05-i-can-make-any-p-value-statistically-significant-with-adaptive-fdr-procedures/

https://www.sciencenews.org/blog/context/experts-issue-warning-problems-p-values

[lupulus]

This is a fascinating thread, which coincidentally I found only after I posted this (provacatively titled) blog entry last night ("Are Homebrew Experiments Scientific?"): https://andybrews.com/2017/02/07/are-homebrew-experiments-scientific/

The tl;dr on it is "Yes, and No," for a variety of reasons (many of them eloquently outlined previously by Brülosophy and Experimental Brewing posts, too). In many cases (in my opinion), the issue is not so much with the experiments or the experimenters--it is how outside readers interpret the experiments, as echoed by others in this thread.

Elsewhere in this thread, I saw something along the lines of "Well, it's only homebrewing, and not done in a lab, so it can't be that scientific." (my paraphrase and some creative interpretation, not a direct quote) As a working scientist, I can say that is a potentially incorrect belief. An experiment is only as scientific as its protocol and interpretation, regardless of whether it is done with fancy equipment or in an expensive, shiny room. Most home setups are limited in the kinds of tests they can do (I don't know any homebrewer with a mass spec in the garage, for instance), but hey, that's why you send stuff out for analysis. If a homebrewer is careful, sets up testable hypotheses, documents their process in detail, interprets the results cautiously, and revises their interpretations in response to constructive review, they are just as scientific as a fancy lab (and quite frankly, sometimes more scientific).

I completely agree with you Andy. I have worked in science for 25 years, lab setting for 7, and clinical research for the last 18, and know that "labs" are not much different from a homebrew setting.

The humble advice to Denny and the IGORs (sounds like Benny and the Jets  )
- Researchers must do a literature review to the extent that is possible, so it is clear in your mind how your experiment will add to the knowledge in the area (it is not that it was never done, but that it must be done for every experiment)
- Researchers must provide an objective evaluation of the quality of the beer whenever possible. Less flaws mean less statistical noise. Flaws in the beer may explain why a difference could not be detected.
- Researchers must maximize the odds of finding a difference should there be one (meaning, pick the best style). The argument that a more flavorful beer is more "real world"is not invalid, but research-wise, one does the experiment with the best chance first, and then does the extrapolation experiment.

Welcome to the forum, Andi !!

[Andy Farke]

<snip>
Please do.  Knowledge is power!  And thank you for your input thus far.  Cheers.

http://simplystatistics.org/2015/08/19/p-0-05-i-can-make-any-p-value-statistically-significant-with-adaptive-fdr-procedures/

https://www.sciencenews.org/blog/context/experts-issue-warning-problems-p-values
[/quote]

Excellent summaries! I've run across a few other posts in the past year or two on the same issue, and will see if I can track them down.

I suppose this is the part where the Bayesians start coming out of the woodwork. (sorry, bad science insider reference...)

[denny]

stats.org used to have an excellent article, but I can't locate it any more.