Membership questions? Log in issues? Email info@brewersassociation.org

Author Topic: Experiments, beer experiments and statistics  (Read 7781 times)

Offline dmtaylor

  • Official Poobah of No Life. (I Got Ban Hammered by Drew)
  • *********
  • Posts: 4730
  • Lord Idiot the Lazy
    • YEAST MASTER Perma-Living
Re: Experiments, beer experiments and statistics
« Reply #30 on: January 31, 2017, 01:55:46 pm »
I'd love it if Brülosophy gave the actual p value in each experiment, so we could come to our own conclusion. Right now it comes across almost like true/false and it's actually way more complex than that.

Actually they do.  Take another look.
Dave

The world will become a much more pleasant place to live when each and every one of us realizes that we are all idiots.

Offline erockrph

  • I must live here
  • **********
  • Posts: 7795
  • Chepachet, RI
    • The Hop WHisperer
Re: Experiments, beer experiments and statistics
« Reply #31 on: January 31, 2017, 02:35:14 pm »
I'd love it if Brülosophy gave the actual p value in each experiment, so we could come to our own conclusion. Right now it comes across almost like true/false and it's actually way more complex than that.

Actually they do.  Take another look.
Missed that, thanks!
Eric B.

Finally got around to starting a homebrewing blog: The Hop Whisperer

Offline homebrewdad7

  • Assistant Brewer
  • ***
  • Posts: 218
    • Homebrew Dad
Re: Experiments, beer experiments and statistics
« Reply #32 on: January 31, 2017, 02:48:58 pm »
I dearly love Brulosophy.  As a guy who does his own website, I will not hesitate to tell you that Brulosophy publishes some of the most useful, innovative content for brewers on the internet.  Every Amazon purchase I make goes to their affiliate link.

With that said, I treat all of the exBEERiments - especially those with no positive finding - with a big grain of salt.  Which, incidentally, is the approach that Marshall always has (and continues to) advocate.

My biggest case in point - the idea that fermentation temperature doesn't matter.  Now, to their credit, since the first such exBEERiment, Brulosophy has done several more, and have actually found it to matter a couple of times.

Now, I don't tend to use the ultra clean kolsch yeasts and the like; I prefer British and Belgian strains, those that have more "character".  There is so much brewing literature (including that from the yeast labs themselves) that talk about temperature being a vital factor in getting certain types of character from these yeasts.  I have done my own experiments at home four times now, using more expressive yeasts, and I have yet to not be able to easily pick out the warm fermented beer in a blind triangle... and when I share it, have had similar results. 

Of course, my process isn't as controlled as that at Brulosophy.  But when all is said and done, while I can agree that temperature control may not be as big a deal to every yeast as we all seem to think, I'm nowhere near prepared to unplug my temperature controller - like almost every brewer I've ever met, I discovered a huge improvement in my beer quality when I started controlling ferm temps. 

Use experiments as data points, use them to help further your own education, but don't make a broad-reaching conclusion and paint it across all of your brewing.  None of the creators of these want you to do that, and you do yourself a disservice if you do.

Offline denny

  • Administrator
  • Retired with too much time on my hands
  • *****
  • Posts: 27137
  • Noti OR [1991.4, 287.6deg] AR
    • Dennybrew
Re: Experiments, beer experiments and statistics
« Reply #33 on: January 31, 2017, 02:53:36 pm »
Every Amazon purchase I make goes to their affiliate link.

Hey, we have one of those, too!  ;)
Life begins at 60.....1.060, that is!

www.dennybrew.com

The best, sharpest, funniest, weirdest and most knowledgable minds in home brewing contribute on the AHA forum. - Alewyfe

"The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts." - Bertrand Russell

Offline denny

  • Administrator
  • Retired with too much time on my hands
  • *****
  • Posts: 27137
  • Noti OR [1991.4, 287.6deg] AR
    • Dennybrew
Re: Experiments, beer experiments and statistics
« Reply #34 on: January 31, 2017, 02:53:59 pm »
I dearly love Brulosophy.  As a guy who does his own website, I will not hesitate to tell you that Brulosophy publishes some of the most useful, innovative content for brewers on the internet.  Every Amazon purchase I make goes to their affiliate link.

With that said, I treat all of the exBEERiments - especially those with no positive finding - with a big grain of salt.  Which, incidentally, is the approach that Marshall always has (and continues to) advocate.

My biggest case in point - the idea that fermentation temperature doesn't matter.  Now, to their credit, since the first such exBEERiment, Brulosophy has done several more, and have actually found it to matter a couple of times.

Now, I don't tend to use the ultra clean kolsch yeasts and the like; I prefer British and Belgian strains, those that have more "character".  There is so much brewing literature (including that from the yeast labs themselves) that talk about temperature being a vital factor in getting certain types of character from these yeasts.  I have done my own experiments at home four times now, using more expressive yeasts, and I have yet to not be able to easily pick out the warm fermented beer in a blind triangle... and when I share it, have had similar results. 

Of course, my process isn't as controlled as that at Brulosophy.  But when all is said and done, while I can agree that temperature control may not be as big a deal to every yeast as we all seem to think, I'm nowhere near prepared to unplug my temperature controller - like almost every brewer I've ever met, I discovered a huge improvement in my beer quality when I started controlling ferm temps. 

Use experiments as data points, use them to help further your own education, but don't make a broad-reaching conclusion and paint it across all of your brewing.  None of the creators of these want you to do that, and you do yourself a disservice if you do.

Well said, Olan.
Life begins at 60.....1.060, that is!

www.dennybrew.com

The best, sharpest, funniest, weirdest and most knowledgable minds in home brewing contribute on the AHA forum. - Alewyfe

"The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts." - Bertrand Russell

The Beerery

  • Guest
Experiments, beer experiments and statistics
« Reply #35 on: January 31, 2017, 05:16:40 pm »
I agree. The triangles are a pain to organize and takes a lot of work. We had (and still have) people screaming at us for more triangle tests. After I provided the one I did using a very common website as basically carbon copy of the tests carried out.  I had the one party I modeled it after kick and scream that I did it wrong, and some other party's ( even ones who screamed for it) say it's all subjective any ways. Sometimes you just can't win! 


Sent from my iPhone using Tapatalk

Offline erockrph

  • I must live here
  • **********
  • Posts: 7795
  • Chepachet, RI
    • The Hop WHisperer
Re: Experiments, beer experiments and statistics
« Reply #36 on: January 31, 2017, 05:49:48 pm »
I agree. The triangles are a pain to organize and takes a lot of work. We had (and still have) people screaming at us for more triangle tests. After I provided the one I did using a very common website as basically carbon copy of the tests carried out.  I had the one party I modeled it after kick and scream that I did it wrong, and some other party's ( even ones who screamed for it) say it's all subjective any ways. Sometimes you just can't win! 

Unfortunately, it's damn near impossible to "prove" something well enough to satisfy all parties, especially when there are tasting panels involved. Which makes it particularly tough to change someone's opinion if they're firmly entrenched in it, since they can take it as carte blanche to dismiss results they don't want to see.
Eric B.

Finally got around to starting a homebrewing blog: The Hop Whisperer

Offline HoosierBrew

  • I must live here
  • **********
  • Posts: 13031
  • Indianapolis,IN
Re: Experiments, beer experiments and statistics
« Reply #37 on: January 31, 2017, 06:10:24 pm »
Unfortunately, it's damn near impossible to "prove" something well enough to satisfy all parties, especially when there are tasting panels involved. Which makes it particularly tough to change someone's opinion if they're firmly entrenched in it, since they can take it as carte blanche to dismiss results they don't want to see.


This, for sure.
Jon H.

Offline lupulus

  • Brewer
  • ****
  • Posts: 391
  • Think like a proton, stay positive!
Re: Experiments, beer experiments and statistics
« Reply #38 on: January 31, 2017, 06:33:04 pm »
Wonderful comments everyone  :)
Indeed, it takes quite a bit of work to do what Brü, and Denny/ Drew/ IGORs do, from the design, to the experiment, to the analysis.
Even if not explicit, the premise of the experiments we have seen so far is to test for differences, not for similarities; so a humble suggestion is to test the variable in a beer that would maximize the differences should the differences exist.
A second suggestion is to get some metric of whether the beer was excellent, very good or good. I am sure we can all agree that in a beer with no or very minor flaws, it would be easier to detect a difference. I understand this is not an easy challenge, but getting 2-3 judges to score the beer (style the brewer intented), would give the readers an idea on whether the "statistical noise" was low, medium or high.
Thoughts, comments, ideas?
“The good thing about science is that it's true whether or not you believe in it.”  Neil deGrasse Tyson

Offline kramerog

  • Brewmaster General
  • *******
  • Posts: 2262
    • My LinkedIn page
Re: Experiments, beer experiments and statistics
« Reply #39 on: January 31, 2017, 10:08:57 pm »
[Rant on]I have 3 issues with many Brulosophy experiments. One many experiments are not based on what is known in the brewing literature so the experiment is sometimes not properly designed and sometimes the experiment does not create knowledge. Two the beers being brewed aren't good candidates for the experiment echoing what lupulus wrote, I.e. a beer with strong flavor is not useful for testing subtle differences. Third the write-ups are horrible as useful details are mixed in with useless details. An executive summary would really help. Pictures of thermometers have little value. [Rant off]

I wonder if the P value  criteria changes based on whether the result is a best practice as opposed to "true" knowledge.

Anyway, taste tests are not the end all of brewing.  The saison tests of experimental brewing doesn't require a taste test to produce useful info. [Still waiting for the write-up Denny and Drew!]



Sent from my XT1095 using Tapatalk

Offline bboy9000

  • Brewmaster
  • *****
  • Posts: 703
  • KCMO
Re: Experiments, beer experiments and statistics
« Reply #40 on: January 31, 2017, 10:43:15 pm »
I like Brulosophy and Experimental Brewing.  The experiments generate more questions than answers but that's science.
Brian
mobrewer

Offline reverseapachemaster

  • Official Poobah of No Life. (I Got Ban Hammered by Drew)
  • *********
  • Posts: 3784
    • Brain Sparging on Brewing
Re: Experiments, beer experiments and statistics
« Reply #41 on: February 01, 2017, 09:27:53 am »
[Rant on]I have 3 issues with many Brulosophy experiments. One many experiments are not based on what is known in the brewing literature so the experiment is sometimes not properly designed and sometimes the experiment does not create knowledge. Two the beers being brewed aren't good candidates for the experiment echoing what lupulus wrote, I.e. a beer with strong flavor is not useful for testing subtle differences. Third the write-ups are horrible as useful details are mixed in with useless details. An executive summary would really help. Pictures of thermometers have little value. [Rant off]

While I share these criticisms I also recognize that at the end of the day these are just a group of homebrewers in very typical homebrewing environments brewing typical homebrew beers while looking at common brewing variables. I agree that an IPA is not a good recipe to explore a small flavor component on a broad scale, if you brew a lot of IPAs then their data point might be useful to your IPAs.

Sometimes they present the findings too broadly for the experiment but IMO the bigger problem is people taking their data points and expanding them to conclusive proof of the entirety of the variable explored.
Heck yeah I blog about homebrewing: Brain Sparging on Brewing

Offline lupulus

  • Brewer
  • ****
  • Posts: 391
  • Think like a proton, stay positive!
Re: Experiments, beer experiments and statistics
« Reply #42 on: February 01, 2017, 10:03:00 am »
[Rant on]I have 3 issues with many Brulosophy experiments. One many experiments are not based on what is known in the brewing literature so the experiment is sometimes not properly designed and sometimes the experiment does not create knowledge. Two the beers being brewed aren't good candidates for the experiment echoing what lupulus wrote, I.e. a beer with strong flavor is not useful for testing subtle differences. Third the write-ups are horrible as useful details are mixed in with useless details. An executive summary would really help. Pictures of thermometers have little value. [Rant off]

While I share these criticisms I also recognize that at the end of the day these are just a group of homebrewers in very typical homebrewing environments brewing typical homebrew beers while looking at common brewing variables. I agree that an IPA is not a good recipe to explore a small flavor component on a broad scale, if you brew a lot of IPAs then their data point might be useful to your IPAs.

Sometimes they present the findings too broadly for the experiment but IMO the bigger problem is people taking their data points and expanding them to conclusive proof of the entirety of the variable explored.
When designing experiments, the typical conditions argument comes up often. Most researchers end up using the design that maximizes the odds of finding a difference, should there be one. There is little value in an experiment that fails to find a difference as you cannot infer a similarity either.  Denny is correct in stating that if many experiments with identical design fail to find a difference, it is more likely that there is no difference but a much larger sample size is needed and the similarity hypothesis is not easy to test (statistically).

Sent from my SM-G930T using Tapatalk

“The good thing about science is that it's true whether or not you believe in it.”  Neil deGrasse Tyson

Offline erockrph

  • I must live here
  • **********
  • Posts: 7795
  • Chepachet, RI
    • The Hop WHisperer
Re: Experiments, beer experiments and statistics
« Reply #43 on: February 01, 2017, 10:42:31 am »
[Rant on]I have 3 issues with many Brulosophy experiments. One many experiments are not based on what is known in the brewing literature so the experiment is sometimes not properly designed and sometimes the experiment does not create knowledge. Two the beers being brewed aren't good candidates for the experiment echoing what lupulus wrote, I.e. a beer with strong flavor is not useful for testing subtle differences. Third the write-ups are horrible as useful details are mixed in with useless details. An executive summary would really help. Pictures of thermometers have little value. [Rant off]

While I share these criticisms I also recognize that at the end of the day these are just a group of homebrewers in very typical homebrewing environments brewing typical homebrew beers while looking at common brewing variables. I agree that an IPA is not a good recipe to explore a small flavor component on a broad scale, if you brew a lot of IPAs then their data point might be useful to your IPAs.

Sometimes they present the findings too broadly for the experiment but IMO the bigger problem is people taking their data points and expanding them to conclusive proof of the entirety of the variable explored.
When designing experiments, the typical conditions argument comes up often. Most researchers end up using the design that maximizes the odds of finding a difference, should there be one. There is little value in an experiment that fails to find a difference as you cannot infer a similarity either.  Denny is correct in stating that if many experiments with identical design fail to find a difference, it is more likely that there is no difference but a much larger sample size is needed and the similarity hypothesis is not easy to test (statistically).

If that's what you're shooting for, then I agree completely. FWIW, I think there is validity to the end point of "would the average beer drinker notice a difference", which is what the Brulosophy experiments are geared towards. It is just less applicable to my brewing.

I've also wondered that if a difference is subtle, then there may be other options that may be more sensitive than a triangle test. If each drinker were asked to give each beer a score from 1-10 rather than comparing them to the other beers, then maybe you might see a trend jump out.
Eric B.

Finally got around to starting a homebrewing blog: The Hop Whisperer

Offline kramerog

  • Brewmaster General
  • *******
  • Posts: 2262
    • My LinkedIn page
Re: Experiments, beer experiments and statistics
« Reply #44 on: February 01, 2017, 05:31:37 pm »
There is a test involving tasting of 5 beers in which the taster has to identify the 2 that are different.  It is mathematically appealing because the chances of randomly guess the 2 odd ones is 1/10 rather than 1/3.  But appears not to be used much because of taste fatigue.

More beers doesn't address the problem of tasting beers with other dissimilarities like infections or different gravities.

Sent from my XT1095 using Tapatalk