+1 for the spray bottle of star san.
If it's a used keg, try replacing all of your gaskets, which is cheap and easy to do. You can also replace your bleed valve cheaply too. I've also gotten a few extra lids for about $12. Also, try some keg lube, which will help give you a tight seal on your lids and posts. Just put some on a paper towel and spread some on your gaskets.
I've always just left my CO2 on and set to 10-12 psi. I could be wrong, but I believe that once the CO2 has filled the keg and carbed the beer, it only pushes CO2 when you create extra space in the keg by dispensing the beer. Could be wrong and would be interesting to see if there is any difference with leaving it on or taking it off.