First off, lets get some terminology down. It's "trub" or "break material" ....those are the technical terms we use as brewers to describe the "sludge".
My experience in the screen will clog with pellet hops, but not with whole hops.
Let me give you my scenario - I use an immersion chiller with a wort pump (see http://www.mrmalty.com/chiller.php
) that circulates the wort while it is chilling and creates a whirlpool which causes the hops to form a cone in the center. I can't use whole hops with this method because they clog. So I normally use pellets and or use another chilling method when using whole hops.
After the wort is chilled I just let the whole thing sit for about 20 minutes to let everything settle. Then, when I run off I get a small amount of hops and trub for the first few seconds and then it run perfectly clear. Near the end it will start to get murky again and I run for a few moments and then shut off. I always compensate for this by brewing 12 gallons and plan on collecting about 11 (leaving 1 gallon behind.) I then use a little less than 1 gallon in yeast and trub when I tranfer to the kegs so collect about 10.
Anyway, you can do this same method by stirring the wort during chilling to form a whirlpool, most of the hops and trub will form a cone in the center. This is assuming you have a valve on the bottom of your kettle to run your wort off the bottom. If you don't you need to get one. Makes like a lot easier. Also, if you are brewing 5 gallon batches, target 5.5 or 6 so that you can leave as much of the hop and trub behind as possible (and plan on losing some wort in the process). But finally, don't stress too much about getting break material and hops in teh fermenter. Leave as much behind as possible but don't stress if some get through.
You can get very clear beer with pellet hops, in fact it really doesn't matter how many hops get into the finished beer - they will settle out and won;t affect clarity of flavor.