I don't know that I even see the time as a cost, at least monetarily. It's an opportunity cost, in that I could be doing something else with my time, but since I'd rather be brewing it's also a benefit (not to mention the great smell of malt and hops that fills the house). I'm brewing in off hours (weekends and evenings) when I'm not getting paid anyway. Could I be raking leaves? Working out? Sure, but I enjoy brewing a whole lot more.
Most of my equipment (and I don't really have anything fancy, so that's a consideration) would have been amortized in the 90's...
Honestly, every time I spend $50 for beer at the liquor store I think, damn I could've brewed 5 gallons for that! Add in (or subtract) re-using yeast and I think homebrewing is pretty darn efficient.