Personally, I doubt it. You did exactly what I do and it doesn't cause me any problems. Second, there's evidence that as long as the starter is cooler than the wort, it may even be beneficial. Obviously I can't say for certain, but I really doubt that was the problem.