I think the correct thing to do would be to do a weighted average, based on the relative amounts of each. If you have 1 gallon of beer with IBU = A and 2 gallons of beer with IBU B then mixing them would give IBU = (1*A + 2*B)/3. If you take that and add it to 3.5 gallons of water with IBU = 0, then the result would be IBU=(1*A +2*B+3.5*0)/6.5
That is the math behind it, but how your beer tastes depends on a lot of other factors.