Author Topic: The t test, p values, and a bit of history  (Read 1647 times)

Offline Wilbur

  • Brewer
  • ****
  • Posts: 396
    • View Profile
The t test, p values, and a bit of history
« on: August 22, 2017, 01:40:45 PM »
An interesting read over at quartz focused on p values and the t test. Researchers are proposing using 0.005 instead of 0.05 as the standard goal for a p value to fix a lack of repeatability in research. Certainly make a lot of Brulosophy, Experimental Brewing, etc. tests "non significant", although I'd never be able to convince people at work to use 0.005 instead anyway. A fun little insight into beer/Guinness history anyway.

https://qz.com/1055287/an-error-made-in-1925-led-to-a-crisis-in-modern-science-now-researchers-are-joining-to-fix-it/

Offline dbeechum

  • Administrator
  • I spend way too much time on the AHA forum
  • *****
  • Posts: 2714
  • Pasadena, CA
    • View Profile
    • Experimental Brewing
Re: The t test, p values, and a bit of history
« Reply #1 on: August 22, 2017, 03:26:30 PM »
I read that last week and I don't know if I'd feel comfortable, in our conditions, moving to that much lower a standard. I also feel like the homebrew community use of the p-value is at least somewhat close to Gosset's original use case.
Drew Beechum - Maltosefalcons.com
- Vote in the AHA GC Election! - http://bit.ly/1aV9GVd  -
-----
Burbling:
Gnome is in the Details
*Experimental Brewing - The Book*
Tap:
Peanut Butter Jelly Time
Tupelo Mead
Farmhouse Brett Saison

Online dmtaylor

  • Official Poobah of No Life.
  • *
  • Posts: 3385
  • Two Rivers, WI
    • View Profile
    • Manty Malters - Meet the Malters! - Dave Taylor
Re: The t test, p values, and a bit of history
« Reply #2 on: August 22, 2017, 04:03:17 PM »
This makes sense only if you can get like 500-1000 people sampling each experiment.  For those who can only get 20-30 people, a p value of like 0.1-0.2 would actually be more reasonable IMHO, and then you can say "well we truly can only place like 80-90% confidence in our results, due to such a low population of samplers... many more experiments with higher populations will be necessary to help support or refute results".
Dave

The world will become a much more pleasant place to live when each and every one of us realizes that we are all idiots.

Offline hackrsackr

  • Assistant Brewer
  • ***
  • Posts: 134
    • View Profile
Re: The t test, p values, and a bit of history
« Reply #3 on: August 24, 2017, 09:43:16 AM »
+10 ^


Sent from my iPhone using Tapatalk

Offline denny

  • Administrator
  • Retired with too much time on my hands
  • *****
  • Posts: 19752
  • Noti OR [1991.4, 287.6deg] AR
    • View Profile
    • Dennybrew
Re: The t test, p values, and a bit of history
« Reply #4 on: August 24, 2017, 03:55:33 PM »
I'm leaning toward ignoring p value for citizen science experiments.
Life begins at 60.....1.060, that is!

www.dennybrew.com

The best, sharpest, funniest, weirdest and most knowledgable minds in home brewing contribute on the AHA forum. - Alewyfe

"The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts." - Bertrand Russell

Offline Andy Farke

  • Assistant Brewer
  • ***
  • Posts: 144
  • Homebrewing Paleontologist
    • View Profile
    • Andy's Brewing Blog
Re: The t test, p values, and a bit of history
« Reply #5 on: August 25, 2017, 03:27:14 PM »
I'm leaning toward ignoring p value for citizen science experiments.

I lean towards keeping p-value, with better education and emphasis of what it does or doesn't mean, and frequent reminders of the various caveats associated with any experiment. Although p-values are not the end-all, be-all of science, they are a consistent way to interpret numerical results. I would argue that results from any experiment are virtually meaningless without some form of quantification. Now the flip side of that is just because you throw a number on it doesn't mean that number means anything....but I also think that if we aren't really willing to take homebrew experiments seriously, they're not really providing anything useful. We're then back to the school of "meh, I'll do what I want" which I guess works in some cases but can result in a lot of wasted time, money, and beer. Which is really why I (as a homebrewer and as a scientist) find homebrew experiments useful--I want to evaluate if step A or ingredient Z or equipment R is worth the trouble and expense!

So where should you ignore (or be extremely cautious with p-value)? If a power test says your sample size ain't big enough to reliably detect a difference, you're probably safe to ignore. Which to me means just conduct more experiments or recruit more tasters!
« Last Edit: August 25, 2017, 03:47:00 PM by Andy Farke »
____________________________
Andy Farke, Homebrewer and Paleontologist
Website: http://www.andybrews.com
Twitter: @andyfarke
Facebook: Farke Brewing

Offline denny

  • Administrator
  • Retired with too much time on my hands
  • *****
  • Posts: 19752
  • Noti OR [1991.4, 287.6deg] AR
    • View Profile
    • Dennybrew
Re: The t test, p values, and a bit of history
« Reply #6 on: August 25, 2017, 04:17:42 PM »
I'm leaning toward ignoring p value for citizen science experiments.

I lean towards keeping p-value, with better education and emphasis of what it does or doesn't mean, and frequent reminders of the various caveats associated with any experiment. Although p-values are not the end-all, be-all of science, they are a consistent way to interpret numerical results. I would argue that results from any experiment are virtually meaningless without some form of quantification. Now the flip side of that is just because you throw a number on it doesn't mean that number means anything....but I also think that if we aren't really willing to take homebrew experiments seriously, they're not really providing anything useful. We're then back to the school of "meh, I'll do what I want" which I guess works in some cases but can result in a lot of wasted time, money, and beer. Which is really why I (as a homebrewer and as a scientist) find homebrew experiments useful--I want to evaluate if step A or ingredient Z or equipment R is worth the trouble and expense!

So where should you ignore (or be extremely cautious with p-value)? If a power test says your sample size ain't big enough to reliably detect a difference, you're probably safe to ignore. Which to me means just conduct more experiments or recruit more tasters!

I guess the older I get and the more experiments I do, the less I think of it as "real" science.  So in that regard, I question the value of the p value.
Life begins at 60.....1.060, that is!

www.dennybrew.com

The best, sharpest, funniest, weirdest and most knowledgable minds in home brewing contribute on the AHA forum. - Alewyfe

"The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts." - Bertrand Russell

Offline chumley

  • Brewmaster
  • *****
  • Posts: 976
    • View Profile
Re: The t test, p values, and a bit of history
« Reply #7 on: August 26, 2017, 03:20:23 AM »
Like Denny, I am old, but unlike Denny, I do not screw around or care about experiments.

However, when I hit a home run, I am willing to share the details.

BREW PACMAN'S BLACK WIDOW KOLSCH, ONLY SWAP OUT THE HONEY MALT FOR MELANOIDIN.

If one brewer pays attention to the above, then my work is done.

Offline BrewBama

  • Senior Brewmaster
  • ******
  • Posts: 1676
    • View Profile
Re: The t test, p values, and a bit of history
« Reply #8 on: August 26, 2017, 09:58:01 PM »
Like Denny, I am old, but unlike Denny, I do not screw around or care about experiments.

However, when I hit a home run, I am willing to share the details.

BREW PACMAN'S BLACK WIDOW KOLSCH, ONLY SWAP OUT THE HONEY MALT FOR MELANOIDIN.

If one brewer pays attention to the above, then my work is done.

For reference: http://www.skotrat.com/skotrat/recipes/ale/kolsch/recipes/14.html


Sent from my iPad using Tapatalk
Huntsville AL