I've had a couple (long, 4+ week) fermentations were the brown scum was pretty well dried on. After a long soak with Alkaline Brewer's Wash I still had to break out the carboy brush to get it all off. So, not that aggressive, but I'd be concerned with the effects of a stiff bristled brush on plastic over time.
By the way, I'm not calling the beer infected. I'm thinking more along the lines of brewers yeast from a different batch staying behind.