I only brew on Friday or Saturday nights, and I start at 7 which is when the baby goes to bed (at least she did until a couple of weeks ago). That way my wife doesn't have to be the sole caregiver for the 6 hours it takes me to get a batch done.
It's a little rough, because it means I don't get to bed until around 1 in the morning, and I usually get up around 6:30.
Before we had a baby, I would start around 10 or 11 in the morning.
I'm actually brewing right now. It's a little after 10 p.m. here, and I've got an hour left to go on the boil.