I've only done a handful of starters, because I generally brew smaller batches. I don't have a stir plate or anything like that, so I try to give myself a week. I let fermentation go for 2-3 days at room temp, then cold crash for another 3-4 days or so. It's probably longer than I need, but it gives me peace of mind.
Followup question for those with more experience than me - do you find you need more time to cold crash certain yeast strains (like belgians, weizens or lagers)?