Author Topic: Experiments, beer experiments and statistics  (Read 3094 times)

Offline lupulus

  • Assistant Brewer
  • ***
  • Posts: 118
  • Think like a proton, stay positive!
    • View Profile
Experiments, beer experiments and statistics
« on: January 30, 2017, 09:59:53 PM »
I wish to share a thought with the forum after reading for the nth time that somebody interprets the researchers not finding a significant difference as the treatments being the same, or that it makes no difference whether I do X vs if I do Y, because this experiment found no difference.
It seems important then to state that failing to find a significant difference can be related to many other factors, for example sample size not being large enough, confounding variables having a larger weight vs the experimental variable, etc.
In other words, failing to prove that there is a 95% probability that treatments are different, does NOT mean that there is a 95% probability that treatments are similar, or even 50%, or any probability. We just do not know...


“The good thing about science is that it's true whether or not you believe in it.”  Neil deGrasse Tyson

Offline dmtaylor

  • Official Poobah of No Life.
  • *
  • Posts: 3224
  • Two Rivers, WI
    • View Profile
Re: Experiments, beer experiments and statistics
« Reply #1 on: January 30, 2017, 10:11:34 PM »
Good point.

I'll again repeat (ad nauseum) that in my opinion, casual experiments run by homebrewers should only be aiming for like 80-85% confidence, not 95%.  In a non-laboratory setting, it's just way too easy for us to screw it up.  In other words, if we were to find, say 80% likelihood that maybe a variable makes a difference, I would have about 80% confidence that that were true..... never 95% confidence, not in an experiment run by homebrewers.

Someone will correct me where I'm wrong, and that's fine.  I'm okay with being wrong.  Cheers all.
Dave

The world will become a much more pleasant place to live when each and every one of us realizes that we are all idiots.

Offline denny

  • Administrator
  • Retired with too much time on my hands
  • *****
  • Posts: 19395
  • Noti OR [1991.4, 287.6deg] AR
    • View Profile
    • Dennybrew
Re: Experiments, beer experiments and statistics
« Reply #2 on: January 30, 2017, 11:29:10 PM »
Something that goes along with this....the key to science is repeatability.  If you do an experiment once, you have a data point.  If it gets done repeatedly, by different people, and most of them get the same result, then you may be on to something.  That's why I always encourage people to try things themselves.  It's also why at EB we have a team of IGORs, so that we have multiple brewers each holding tastings with multiple people.
Life begins at 60.....1.060, that is!

www.dennybrew.com

The best, sharpest, funniest, weirdest and most knowledgable minds in home brewing contribute on the AHA forum. - Alewyfe

"The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts." - Bertrand Russell

Offline klickitat jim

  • I must live here
  • **********
  • Posts: 8460
    • View Profile
Re: Experiments, beer experiments and statistics
« Reply #3 on: January 30, 2017, 11:58:47 PM »
Let's not forget that this forum is unique in that it is impossible to 100% prove anything here. To attempt to do so is both folly and honorable. It's what we do. And it's not necessarily a bad thing. After all, our end product is ultimately judged entirety subjectively

Offline IPAnic

  • Cellarman
  • **
  • Posts: 31
    • View Profile
Re: Experiments, beer experiments and statistics
« Reply #4 on: January 31, 2017, 12:11:56 AM »
Let's not forget that this forum is unique in that it is impossible to 100% prove anything here. To attempt to do so is both folly and honorable. It's what we do. And it's not necessarily a bad thing. After all, our end product is ultimately judged entirety subjectively
This

Offline denny

  • Administrator
  • Retired with too much time on my hands
  • *****
  • Posts: 19395
  • Noti OR [1991.4, 287.6deg] AR
    • View Profile
    • Dennybrew
Re: Experiments, beer experiments and statistics
« Reply #5 on: January 31, 2017, 12:39:58 AM »
Let's not forget that this forum is unique in that it is impossible to 100% prove anything here. To attempt to do so is both folly and honorable. It's what we do. And it's not necessarily a bad thing. After all, our end product is ultimately judged entirety subjectively
This

DOUBLE this!
Life begins at 60.....1.060, that is!

www.dennybrew.com

The best, sharpest, funniest, weirdest and most knowledgable minds in home brewing contribute on the AHA forum. - Alewyfe

"The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts." - Bertrand Russell

Offline kramerog

  • Senior Brewmaster
  • ******
  • Posts: 1810
    • View Profile
    • My LinkedIn page
Re: Experiments, beer experiments and statistics
« Reply #6 on: January 31, 2017, 01:55:46 AM »
Much of what I get out of the various experiments is that is really hard to run a proper experiment particularly one with a triangle test.

Sent from my XT1095 using Tapatalk


Offline Philbrew

  • Brewmaster
  • *****
  • Posts: 866
    • View Profile
Re: Experiments, beer experiments and statistics
« Reply #7 on: January 31, 2017, 03:36:17 AM »
Much of what I get out of the various experiments is that is really hard to run a proper experiment particularly one with a triangle test.

Sent from my XT1095 using Tapatalk
All of the above!  And I really love the thread title, Mr. Twain.  ;)
Many of us would be on a strict liquid diet if it weren't for pretzels.

Offline reverseapachemaster

  • Official Poobah of No Life.
  • *
  • Posts: 3136
    • View Profile
    • Brain Sparging on Brewing
Re: Experiments, beer experiments and statistics
« Reply #8 on: January 31, 2017, 03:48:44 AM »
This is very true. Homebrewers cite to a given experiment as conclusive proof of a given point but the conclusion rarely has such a broad application. Generally the results are some evidence of a particular conclusion under a particular set of circumstances.
Heck yeah I blog about homebrewing: Brain Sparging on Brewing

Offline stpug

  • Brewmaster
  • *****
  • Posts: 701
    • View Profile
Re: Experiments, beer experiments and statistics
« Reply #9 on: January 31, 2017, 03:24:09 PM »
This is very true. Homebrewers cite to a given experiment as conclusive proof of a given point but the conclusion rarely has such a broad application. Generally the results are some evidence of a particular conclusion under a particular set of circumstances.

...with a particular set of people's tongues

Offline dmtaylor

  • Official Poobah of No Life.
  • *
  • Posts: 3224
  • Two Rivers, WI
    • View Profile
Re: Experiments, beer experiments and statistics
« Reply #10 on: January 31, 2017, 03:26:35 PM »
Accept all inputs with grains of salt.  Always with grains of salt.  Nobody nobody NOBODY is perfect and knows everything about everything.  NOBODY.
Dave

The world will become a much more pleasant place to live when each and every one of us realizes that we are all idiots.

Offline goschman

  • Official Poobah of No Life.
  • *
  • Posts: 3432
    • View Profile
Re: Experiments, beer experiments and statistics
« Reply #11 on: January 31, 2017, 03:29:47 PM »
Although I am usually very interested in the results I am usually underwhelmed by the amount of data/tasters in any given experiment.
On Tap/Bottled: Haze for Daze IPA, G Pils, Maibock, Watermelon Cider         

Fermenting: Kolsch
Up Next: Summer Ale, Gose

Offline denny

  • Administrator
  • Retired with too much time on my hands
  • *****
  • Posts: 19395
  • Noti OR [1991.4, 287.6deg] AR
    • View Profile
    • Dennybrew
Re: Experiments, beer experiments and statistics
« Reply #12 on: January 31, 2017, 03:46:45 PM »
This is very true. Homebrewers cite to a given experiment as conclusive proof of a given point but the conclusion rarely has such a broad application. Generally the results are some evidence of a particular conclusion under a particular set of circumstances.

Which is my point anout repeatability
Life begins at 60.....1.060, that is!

www.dennybrew.com

The best, sharpest, funniest, weirdest and most knowledgable minds in home brewing contribute on the AHA forum. - Alewyfe

"The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts." - Bertrand Russell

Offline denny

  • Administrator
  • Retired with too much time on my hands
  • *****
  • Posts: 19395
  • Noti OR [1991.4, 287.6deg] AR
    • View Profile
    • Dennybrew
Re: Experiments, beer experiments and statistics
« Reply #13 on: January 31, 2017, 03:48:15 PM »
Although I am usually very interested in the results I am usually underwhelmed by the amount of data/tasters in any given experiment.

Including the ones we do?  What would you consider an adequate number?  I think the fewest brewers we've had was maybe 5 and each of those had around 7 tasters.
Life begins at 60.....1.060, that is!

www.dennybrew.com

The best, sharpest, funniest, weirdest and most knowledgable minds in home brewing contribute on the AHA forum. - Alewyfe

"The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts." - Bertrand Russell

Offline hopfenundmalz

  • Global Moderator
  • I must live here
  • *****
  • Posts: 8844
  • Milford, MI
    • View Profile
Re: Experiments, beer experiments and statistics
« Reply #14 on: January 31, 2017, 03:56:21 PM »
Before I retired I was exposed to GFSS. You Gould spend as much time, or more, on he design of the experiment as doing the experiment.
Jeff Rankert
Ann Arbor Brewers Guild
AHA Governing Committee
BJCP National
Home-brewing, not just a hobby, it is a lifestyle!