I'd think the granite would be plenty cool on it's own. I'd insulate the tower, sure, but I don't think you need continual insulation from the kegerator into the tower.
If you're still concerned about cooking, get a copper rod and use that as a heat sink of sorts, fit what you can into the tower, and run the bulk it down into the kegerator. Should pull the heat out of the tower some, and still get you a narrower than 4" hole.