The only ball valve I have that is of any concern is the one on the boil kettle. I rinse it when cleaning the kettle and let the heat of the next boil kill off the bugs.
You might want to take that ball valve apart once in a while and inspect it. I had one on my boil kettle for years and cleaned it by curculating PBW through it while cleaning the pump, chiller, and kettle. I always thought it got really clean this way until I took it apart and found some gunk around the outside of the ball that had been there for who knows how long. It didn't add any off flavors to the beer since, as you said the boil will kill of everything, but it was disgusting looking. I eventually replaced it with a butterfly valve which is way easier to keep clean.
Regarding the ball valves on the racking ports of my conicals, I sinply put them (in the partially opened position) in the fermenter and clean them while cleaning the concials. I use a spray ball to clean my fermenters and it works well to clean them. The valves are immersed in 3 gallons of PBW and the spraying and pumping action circulates the liquid in the cone and through the valves. I take them apart after every 5 brews or so to inspect them and they are always clean.