I'm with Jeff on this one. Because I can skim that material easily, I do it for all my brews that aren't First Wort Hopped. There is ample evidence in my opinion that the scum on the boil has flavor negative impacts, so it seems that it is worth doing when I'm not going to lose some of my FWHs.
Ample evidence? Could you cite some of that?
For the record I will often skim just for something to do. I don't think its much different than the rest of the protein, and when I don't skim I find that the floating denatured protein winds up on the side of my kettle as the level drops. So I don't think I'm affecting my beer by doing this, just making it a little easier to clean the kettle later.
I do also sometimes see a little hop color (and presumably oils) sticking to the scum when I first add my bittering addition. SIn this respect maybe it has some very minor affect on tthe bitterness.