I routinely let my wort set for several hours before pitching to let it come down to pitching temp. Especially in the summer, and always with lagers that I'm trying to pitch at 46-48F. I rarely let it sit as long as you're talking about but, as stated above, if your sanitation was good, it shouldn't be a problem.
Recently I started wondering just how sanitary my wort was by the time it was chilled and made it to the fermenter. So I've occasionally started pulling a sample before pitching the yeast and putting it in a sanitized jar with foil on top and just letting it sit there in a warm place for days to see if anything starts happening. So far I've let samples sit as long as five days without any type of activity at all.