If you want the nitty-gritty, you can read Palmer's section on it here:
http://www.howtobrew.com/section1/chapter5-5.htmlIt includes a handy nomograph to use for calculations.
The hop utilization numbers Palmer got from data taken by Tinseth. If it were me, unless I was going for something very specific in terms of bitterness, I would just toss the hops in and go with the higher IBUs. If I really wanted to try to correct for it, I would take a rough stab at it by looking up the utilization for the recipe pre-boil gravity and ratio that with the utilization for a full boil gravity. For instance, say your recipe has you boiling 3 gallons of 1.080 wort for 60 minutes. From the table that would be a utilization of 0.176. Now you want to boil 5 gallons for 60 minutes, so your pre-boil gravity is 3/5th 1.080 or 1.048, and your new utilization is about 0.230 or so, so you could cut your hops back to about 0.176/0.230 = 76 percent.
Edit: I used "efficiency" when I should have used "utilization" to keep the terminology consistent regarding hop usage