[Iliff Ave]

Although I am usually very interested in the results I am usually underwhelmed by the amount of data/tasters in any given experiment.

Including the ones we do?  What would you consider an adequate number?  I think the fewest brewers we've had was maybe 5 and each of those had around 7 tasters.

Mostly referring to exbeeriments. When only 12 correct tasters are needed to provide significance, it just doesn't seem like enough data.

Am I correct, in that yours involve multiple individuals brewing separate batches? To me, that throws in a lot of other variables like water, fermentation profiles, OG, FG, etc. I am not that familiar with your process so maybe you control those aspects closely.

EDIT - I am not calling anyone's process into question as I myself am not very scientific. I would be the last person to say anyone is doing it wrong or they should do it another way. Just some basic observations that influence my perspective.

[braufessor]

I agree with all of the above for the most part.... ultimately we are talking about individual data points, under "home-brew conditions" as opposed to "lab conditions."  And -  the "results" are generally being gathered subjectively.  I don't think anyone should be looking at most of these tests/experiments as "published science."

That said - I think that Brulosophy and Denny/Drew seem to always/almost always go out of their way to reinforce that they are not declaring anything "certain" or "factual" - rather simply sharing:  "here is what we did, here is why we did it this way, here is how we evaluated it, here is what we found....... check it out yourself and see what you think."

In that regard - I really enjoy having a bunch of people checking things out and sharing what they find.  It allows me to potentially agree and further feel comfortable with my own observations on a topic.  Or, perhaps it inspires me to check something out further that I was curious about.  Or - maybe it is just something I don't care that much about and I ignore it.

At the end of the day - there will likely never be any one test or observation that leads to an entirely new world of brewing.  However, the cumulative power of people sharing information with each other eventually has the ability to move everyone ahead incrementally over time.

[dmtaylor]

Although I am usually very interested in the results I am usually underwhelmed by the amount of data/tasters in any given experiment.

Including the ones we do?  What would you consider an adequate number?  I think the fewest brewers we've had was maybe 5 and each of those had around 7 tasters.

I'll bet he's referring more to the big B-to-the-y with the umlaut.

EDIT: Ah yes, I see it above now.

[HoosierBrew]

I think experiments/exbeeriments are great for the hobby, as long as the experiment process is explained well, and that we take it as it should be - not as gospel but as a data point. In my mind, enough similar data points performed in a coherent fashion constitute a likelihood in my mind. Not always. I also like reading the posted professional literature and studies as well as a means to decide what I feel is applicable/feasible, or just for the sake of knowing. But what trumps it all for me is using my senses to eval my beer. After all, I don't drink studies or experiments.

[Joe Sr.]

You know what?  I find them interesting to read and to think about.  Sometimes there's useful takeaways.  Do some critical thinking and take away what you will.

Anyone looking to these sorts of experiments for definitive proof of something is looking to hard.

If you don't like how the experiment was set up, disregard it. 

Whatever.  Didn't cost you nothing anyway.

[klickitat jim]

Denny's point on repeatability is probably the most important thing in any attempt to "prove" a home brewing thing. (an home brewing?) Brü tends to do an experiment with one brewer and lots of tasters. Seems to lean toward stronger evidence of a conclusion on that brew. Denny and Drew use multiple brewers who may have less tasters each. That seems to provide stronger evidence regarding repeatability.

So, a conclusion that is based on numerical consensus if fine, but what if I can't repeat it? Scientifically proven data may be the strongest but if I can't detect it in my beer, does it really matter?

Example: a meter might show irrefutable evidence that there is DMS in a beer I boiled 60 min instead of 90, but if it's low enough that I can't detect it does it matter. I say no. But how should I present that info?

I reduced boil time from 90 to 60 and found no DMS so, no one has to boil 90 minutes... ever! As long as you don't care what the instrumentation says, and you boil on my equipment, and have the same sensitivity to DMS that I do.

I really think we'd all be doing ourselves a huge favor by learning how to interpret information and how we might apply it to our own needs, rather than always grabbing our pitch forks and torches.

Grains of salt in other words

[denny]

Although I am usually very interested in the results I am usually underwhelmed by the amount of data/tasters in any given experiment.

Including the ones we do?  What would you consider an adequate number?  I think the fewest brewers we've had was maybe 5 and each of those had around 7 tasters.

Mostly referring to exbeeriments. When only 12 correct tasters are needed to provide significance, it just doesn't seem like enough data.

Am I correct, in that yours involve multiple individuals brewing separate batches? To me, that throws in a lot of other variables like water, fermentation profiles, OG, FG, etc. I am not that familiar with your process so maybe you control those aspects closely.

EDIT - I am not calling anyone's process into question as I myself am not very scientific. I would be the last person to say anyone is doing it wrong or they should do it another way. Just some basic observations that influence my perspective.

Yeah, multiple brewers each having many tasters.  You're right that it can introduce more variables, but we try to control those as tightly as possible.  OTOH, there is a lot of variation between normal homebrewers.  That's why our IBU experiment showed devaitions of +/- 40% from predicted IBUs.  But that's kinda what we wanted to know.

[pete b]

The hardest thing about beer experiments is the amount of work and time it takes to make a batch and the time it takes for them to be finished. Food is generally so much easier. America's Test Kitchen can test scrambled eggs with a variable changed 6 different ways in an hour and taste side by side. Also, the differences we are usually looking in beer are often extremely subtle and ridiculously subjective. Tenderness and fluffiness of scrambled eggs is relatively objective. 

[erockrph]

Although I am usually very interested in the results I am usually underwhelmed by the amount of data/tasters in any given experiment.

Including the ones we do?  What would you consider an adequate number?  I think the fewest brewers we've had was maybe 5 and each of those had around 7 tasters.

Mostly referring to exbeeriments. When only 12 correct tasters are needed to provide significance, it just doesn't seem like enough data.

Am I correct, in that yours involve multiple individuals brewing separate batches? To me, that throws in a lot of other variables like water, fermentation profiles, OG, FG, etc. I am not that familiar with your process so maybe you control those aspects closely.

EDIT - I am not calling anyone's process into question as I myself am not very scientific. I would be the last person to say anyone is doing it wrong or they should do it another way. Just some basic observations that influence my perspective.

I have much of the same reservations as you regarding the sample size over at Brulosophy and the crowdsourced IGOR approach at Experimental brewing. Don't get me wrong, the experiments are always great food for thought, and are generally well-run and planned out. I hope to see them continue for a long time to come as they are great for our hobby. That said, there are a couple of caveats that you have to keep in mind.

Marshall and the rest of the Brulosophy folks took a good approach by using a low p-value for determing statistical significance, given their relatively small sample sizes. This means that if an exbeeriment does return a positive result, you can be reasonably sure that this correlation is not due to chance.

But the flip side is that not finding a significant result is of minimal value. What that result means is that if there is a correlation between the two variables, then the experiment design was not sensitive enough to find it, not necessarily that the null hypothesis is true. This is why so many exbeeriments return no statistical significance. If they fail to find a statistical significance, it is nowhere near as strong of a result as a positive finding.

As far as the IGOR experiments go, the tradeoff is giving up some control over experiment process for additional sample size. I read plenty of bad experiments from trained scientists in medical journals all the time, so I imagine it's really easy for confounding variables to slip in when going through this crowdsourced approach.

None of this is meant as a criticism for either Brulosophy or XB. I think they are both doing a great job - far better than I can do in my kitchen. The caution comes in with the interpretation of the results.

[denny]

I have much of the same reservations as you regarding the sample size over at Brulosophy and the crowdsourced IGOR approach at Experimental brewing. Don't get me wrong, the experiments are always great food for thought, and are generally well-run and planned out. I hope to see them continue for a long time to come as they are great for our hobby. That said, there are a couple of caveats that you have to keep in mind.

Marshall and the rest of the Brulosophy folks took a good approach by using a low p-value for determing statistical significance, given their relatively small sample sizes. This means that if an exbeeriment does return a positive result, you can be reasonably sure that this correlation is not due to chance.

But the flip side is that not finding a significant result is of minimal value. What that result means is that if there is a correlation between the two variables, then the experiment design was not sensitive enough to find it, not necessarily that the null hypothesis is true. This is why so many exbeeriments return no statistical significance. If they fail to find a statistical significance, it is nowhere near as strong of a result as a positive finding.

As far as the IGOR experiments go, the tradeoff is giving up some control over experiment process for additional sample size. I read plenty of bad experiments from trained scientists in medical journals all the time, so I imagine it's really easy for confounding variables to slip in when going through this crowdsourced approach.

None of this is meant as a criticism for either Brulosophy or XB. I think they are both doing a great job - far better than I can do in my kitchen. The caution comes in with the interpretation of the results.

Boy, do I agree!  I think that's why we've only had one experiment where the results were so definite that we could actually draw a conclusion.  Most of the time al we can do is say "here's how it turned out and here's what we think is the reason".  In the current IBU experiment, the results raised more questions than they answered.  And FWIW, we're working on coordinating experiments with Marshall to hopefully get more data out there.

[Joe Sr.]

Boy, do I agree!  I think that's why we've only had one experiment where the results were so definite that we could actually draw a conclusion.  Most of the time al we can do is say "here's how it turned out and here's what we think is the reason".  In the current IBU experiment, the results raised more questions than they answered.  And FWIW, we're working on coordinating experiments with Marshall to hopefully get more data out there.

FWIW, I think there's a ton of value in results that show no definitive proof of anything but a wide variation of results across brewers and systems.

I think that sometimes beginning brewers (and probably advanced brewers, too) think it's like assembling a piece of furniture and all you need to do is follow the instructions and you get what's pictured on the box.  It doesn't quite work that way.

It's just as important as anything to know your system, your process, and what you can expect as results.  Which, perhaps unfortunately for beginning brewers, means it takes some time and multiple batches to start figuring things out.

[klickitat jim]

No doubt Joe! Most people have all of the skills needed, reading, measuring, pouring, stiring... if they would actually stick with the book, they might get their legs under them quicker. Like, actually following How to Brew rather than just reading it and then jumping into inventing your own recipes for crazy invented beer styles.

I don't believe it's vital that everyone know WHY or HOW every aspect works. For example, I use a water calculator and usually adjust Ca, Mg, Na, Cl, and SO4, and acid adjustments, but I only have a super simplistic understanding of Why and How. If the only way one could create tasty beer was to know enough about chemistry to build your own water calculator... I would quit brewing

[denny]

FWIW, I think there's a ton of value in results that show no definitive proof of anything but a wide variation of results across brewers and systems.

I think that sometimes beginning brewers (and probably advanced brewers, too) think it's like assembling a piece of furniture and all you need to do is follow the instructions and you get what's pictured on the box.  It doesn't quite work that way.

It's just as important as anything to know your system, your process, and what you can expect as results.  Which, perhaps unfortunately for beginning brewers, means it takes some time and multiple batches to start figuring things out.

Absolutely!  In tomorrow's episode we talk to 5 of the brewers from the IBU experiment to look at what the differences were and why they might have occurred.

[dmtaylor]

...not finding a significant result is of minimal value. What that result means is that if there is a correlation between the two variables, then the experiment design was not sensitive enough to find it, not necessarily that the null hypothesis is true. This is why so many exbeeriments return no statistical significance. If they fail to find a statistical significance, it is nowhere near as strong of a result as a positive finding.

I like how you phrased this.  This is sort of the point I've been trying to make, and the OP as well if I understood correctly.  Yet I still feel a need also to expound on this: if the tasters are almost never able to reliably detect a difference, then personally I figure, what's the real value of making a conclusion statement to this effect?  What if instead of aiming for these exbeeriments to give a 95% reliable result, we use a p value of like 0.15 or 0.2 which will tell you "well, MAYBE there's something going on here, but more exbeeriments will be needed to try to find out".  Then again, to be fair, maybe it's just easier this way to avoid too much speculation in the conclusion statements.  I guess the argument might be a horse apiece; regardless of the p, we can always say like, wow, we came really close to the 95% confidence, so maybe more xBmts are warranted.  I dunno, I guess I'll be quiet now as we're talking in circles again as we love to do.

None of this is meant as a criticism for either Brulosophy or XB. I think they are both doing a great job - far better than I can do in my kitchen. The caution comes in with the interpretation of the results.

Ditto.  I've got nothing but love and respect for anyone who runs xBmts or IGORs and publishes the results publicly and free of charge!  Thanks all so much -- we and you all know who you are.  Cheers.

[erockrph]

I'd love it if Brülosophy gave the actual p value in each experiment, so we could come to our own conclusion. Right now it comes across almost like true/false and it's actually way more complex than that.


Edit - incorrect