I skim because it's a cooking habit I acquired prior to homebrewing, from decades of making chicken stock. Stand over the stock pot... peer into it... skim gunk into a glass mug, look at it critically... it's part of the process for me, and process is why I brew in the first place. Skimming also has the benefit that I'm watching the boil, so I don't have boilovers -- particularly important because I tend to push the limits on what I can fit into my brew kettle.
While I have no idea if skimming makes the beer any better, I would only stop skimming if I knew it was bad for the end product.