I routinely pull my yeast starters out of a 40-degree fridge, decant and pitch the slurry into 70-degree wort. Never had a problem doing this. Lag times between 4 and 6 hours are normal and terminal gravity is usually reached within 3 to 5 days of pitching, so no evidence of any negative effect.