I've heard that said so so frequently, and it makes so much sense, that I'm going to take one of my favorite all-grain recipes and convert it to a small extract batch--not with pantry leftovers but with good fresh ingredients.
The primary issues that keep me from buying kits (and that in a sense got me into AG brewing) are the quantity and the logistical issues--there are several points in the process where I need to lift and carry a full fermenter, so a little over 3 gallons is my max. I enjoy the recipe-building process as well as the mashing experience, and appreciate the experimentation small batches free me to do, but there are times in the year when I'd be more than happy to skip past the recipe construction, trip to the LHBS, mash, etc., order a kit, and get to the boil. I can get pretty close to that model, thought not quite. (The closest analogy I can think of is my tiramisu: I have made my own ladyfingers, and they're excellent, but I now live somewhere I can buy high-quality ladyfingers, cutting out hours from the production of one of my "company" desserts.)