Hey gwilkosz, I brew all-grain, and at least two-thirds of my batches are 3 - 4 gallons in size. I often brew two 3.5 gallon batches in a day, rather than doing one larger batch. I've even done three-in-a-day, but one must be psyched to pull that off.
The obvious downside is time; it takes very little difference in time to brew 3.5 vs. 5.5. But with two-brews-in-a-day, and with organization and preparation, I really doubt that I spend more time per gallon than when I brew five gallon batches.
The advantages (for the dubious) are variety in your available beer, rather than sheer quantity; cleanup may be easier as your mash tun, kettle, etc., can be smaller in size; easier storage given that your gear can be smaller; easier to brew indoors (kitchen stove, although I usually go outside); less weight to muscle brew around (assuming you don't have all the pumps); and best of all, it's cheap to go stainless steel the whole way (I ferment in kegs and transfer with CO2). I was very happy to say bye-bye to buckets and glass. I didn't have to spend a grand on a conical, and I have something that fits easily in a fridge for fermentation. Also, it's easy to run side-by-side tests as a normal matter of your brewing technique, and easier to store your final product (although 3 gallon kegs are pricey, so mostly I use larger ones).
One big disadvantage is the percent loss of wort increases with smaller batches. That is, due to equipment and methods, you may be used to loosing (for example) a quart or two of wort in the kettle and primary and secondary, etc. Obviously, loosing a quart of three gallons hurts more than loosing a quart of five gallons. I routinely use whirlfloc, and man that can hurt to see a cooled kettle filled with "fluff." I do suggest using just half a whirlfloc on a smaller batch.
There are ways to deal with this loss, and I now lose almost nothing from the kettle (this is likely controversial and will take more words and pictures, will try to update later. Basically, cold filtration works, no HSA problems, and you can always use filtered wort as yeast starters or gyle if you are concerned about flavor impacts).
Recently, as I have more free time in the evenings than during the day, I've started experimenting with mashing in the evening, then brewing the next day. In three beers, one of which is now six months old, I've seen no ill effects. Of course, keeping the wort cold overnight can be an issue (not in MIchigan in January, however). Net result is that by noon the next day, I'm done brewing and doing something else!
Sometimes, my second batch will be a partial mash to save a bit of effort, but with a five gallon cooler, a zapap, and a large rectangular cooler, I can easily do two mashes at once. You can even manage doing a double double-decoction, but that is again, something to be psyched for. I suggest not drinking home brew while attempting this! Sometimes, I'll do a big mash in the large cooler, with a smaller mash in one of the other coolers, and do one wort straight from the large cooler, and the second wort as a blend.
I don't have to worry about the storage space issues, but I did brew something like 15 lagers last winter and it was cool to experiment with all those styles. I'm still drinking six of them. There's no way I would've done all that with five gallon batch sizes.