I actually did a side-by-side brew on this because I like to use hop bags and I wanted to gauge their effects on taste. Two identical brews, except one I boiled using hop bags (the little nylon 8" kind) the other tossing the hops directly in the kettle. I used four hop additions, pellets, whole, relatively unpacked, and packed full. I tried to run the gamut all at once. Then a "panel" of sixteen tasters (well 15, I did it twice) for a double-blind triangle test consisting of 3 BJCP-certified judges, the head brewer from a local micorbrewery, a couple of the guys from the LHBS, assorted fellow homebrewers, and a couple others.
Four out of the sixteen were able to correctly separate the two brews and identify the hop bag brew vs the all natural brew. That's 1 out of 4 vs the 1 out of 6 you would expect just from luck alone, so my conclusion was that there is a difference, especially since 2 out of the 3 BJCP judges got it right (the head brewer did not). I myself got it wrong twice, so I've decided to continue using the hop bags since I primarily brew for myself! Interestingly, and entertainingly, most tasters insisted that the difference was "obvious", that is they did before they got the results that showed they were wrong. Two of my friends faulted the test saying that the beers must be at different temperatures, blah blah. They weren't!
Anyway, I use a giant hop bag for 10 gallon boils in my keggle, but still use the smaller hop bags when I brew smaller batches in my smaller kettle. Makes life easy for me, though I really need to look into making a hop taco...
Anyone out there able to directly measure IBUs? I'd love to repeat the test and get a real measurement.