Similar story on a more compressed time scale. I was initially advised to go big because "most people eventually scale up to ten-gallon batches..."
But I realized pretty quickly that at my rate of consumption, 5-gallon batches are way too big for me to iterate through recipes at a reasonable pace. I switched to 2.5-gallon batches and then ultimately down to 1-gallon (now metricized to 4-liter) batches.
I agree that the brewing is more fun than the drinking anyway, and at 4L, I can brew every other weekend. And at ~1 kg of grain, a few grams of hops, I'm out maybe $10 if I don't like how a recipe turns out. Less than that if I'm repitching yeast.
Someday, when I've settled on my go-to recipes, I assume I'll scale back up to, say, a 10-L standard batch size, but who knows.