[lupulus]

I wish to share a thought with the forum after reading for the nth time that somebody interprets the researchers not finding a significant difference as the treatments being the same, or that it makes no difference whether I do X vs if I do Y, because this experiment found no difference.
It seems important then to state that failing to find a significant difference can be related to many other factors, for example sample size not being large enough, confounding variables having a larger weight vs the experimental variable, etc.
In other words, failing to prove that there is a 95% probability that treatments are different, does NOT mean that there is a 95% probability that treatments are similar, or even 50%, or any probability. We just do not know...

[dmtaylor]

Good point.

I'll again repeat (ad nauseum) that in my opinion, casual experiments run by homebrewers should only be aiming for like 80-85% confidence, not 95%.  In a non-laboratory setting, it's just way too easy for us to screw it up.  In other words, if we were to find, say 80% likelihood that maybe a variable makes a difference, I would have about 80% confidence that that were true..... never 95% confidence, not in an experiment run by homebrewers.

Someone will correct me where I'm wrong, and that's fine.  I'm okay with being wrong.  Cheers all.

[denny]

Something that goes along with this....the key to science is repeatability.  If you do an experiment once, you have a data point.  If it gets done repeatedly, by different people, and most of them get the same result, then you may be on to something.  That's why I always encourage people to try things themselves.  It's also why at EB we have a team of IGORs, so that we have multiple brewers each holding tastings with multiple people.

[klickitat jim]

Let's not forget that this forum is unique in that it is impossible to 100% prove anything here. To attempt to do so is both folly and honorable. It's what we do. And it's not necessarily a bad thing. After all, our end product is ultimately judged entirety subjectively

[IPAnic]

Let's not forget that this forum is unique in that it is impossible to 100% prove anything here. To attempt to do so is both folly and honorable. It's what we do. And it's not necessarily a bad thing. After all, our end product is ultimately judged entirety subjectively

This

[denny]

Let's not forget that this forum is unique in that it is impossible to 100% prove anything here. To attempt to do so is both folly and honorable. It's what we do. And it's not necessarily a bad thing. After all, our end product is ultimately judged entirety subjectively

This

DOUBLE this!

[kramerog]

Much of what I get out of the various experiments is that is really hard to run a proper experiment particularly one with a triangle test.

Sent from my XT1095 using Tapatalk

[Philbrew]

Much of what I get out of the various experiments is that is really hard to run a proper experiment particularly one with a triangle test.

Sent from my XT1095 using Tapatalk

All of the above!  And I really love the thread title, Mr. Twain. 

[reverseapachemaster]

This is very true. Homebrewers cite to a given experiment as conclusive proof of a given point but the conclusion rarely has such a broad application. Generally the results are some evidence of a particular conclusion under a particular set of circumstances.

[stpug]

This is very true. Homebrewers cite to a given experiment as conclusive proof of a given point but the conclusion rarely has such a broad application. Generally the results are some evidence of a particular conclusion under a particular set of circumstances.

...with a particular set of people's tongues

[dmtaylor]

Accept all inputs with grains of salt.  Always with grains of salt.  Nobody nobody NOBODY is perfect and knows everything about everything.  NOBODY.

[Iliff Ave]

Although I am usually very interested in the results I am usually underwhelmed by the amount of data/tasters in any given experiment.

[denny]

This is very true. Homebrewers cite to a given experiment as conclusive proof of a given point but the conclusion rarely has such a broad application. Generally the results are some evidence of a particular conclusion under a particular set of circumstances.

Which is my point anout repeatability

[denny]

Although I am usually very interested in the results I am usually underwhelmed by the amount of data/tasters in any given experiment.

Including the ones we do?  What would you consider an adequate number?  I think the fewest brewers we've had was maybe 5 and each of those had around 7 tasters.

[hopfenundmalz]

Before I retired I was exposed to GFSS. You Gould spend as much time, or more, on he design of the experiment as doing the experiment.