A lot of good comments here. In the "Yeast" book, I think JZ and CW discouraged two types of yeast pitching:
1.) yeast into overly warm wort (it kills the yeast)--so wait until the wort gets down to 80F or less.
2.) warm yeast (e.g., at 60-80 F) into cool/chilled wort (e.g., 40-50 F), as in a lager, as it might shock them into dormancy.
They do encourage pitching the cooler yeast slurry into the warmer wort as it tends to "wake them up for the party," so you should have no problem pitching straight from the fridge into the wort. I do it routinely with my 2 L starters (w/o pouring off the supernatant or "spent" wort) with no observable downside. Vigorous fermentation within 8-12 hours, and no off flavors/aromas in the finished product.
I don't discount the idea to decant the spent wort, however. It does make logical sense. But it hasn't had a noticable effect on any of my 10 gallon batches. If you do it, I would suggest cold-crashing the starter for at least 48 hours (IMO, 12-24 hours is not enough) to allow more time for the yeast to settle and the spent wort to clear. Then, on brew day, you can decant, recover the flask, and let the yeast slurry slowly warm up to ambient (or pitching) temps. I've had starters that haven't appreciably settled enough after 24-36 hours, so I just re-suspended the whole slurry and pitched the whole thing w/o detrimental effects on the beer.