The longer you wait to pitch the yeast the more chance there is for critters to get a footing and funkify the beer. You will likely be ok if it's only a couple of hours, but you really want fermentation to get going as soon as possible after the wort is cooled from the boil to avoid infection. I don't know exactly how long is *too long*, you'll probably be ok if it's within the same day. Keep it held at fermentation temp, I can't see a reason colder would really help.
Always potential for an interesting beer if it's a style that can handle funk.