Absolutely, I agree.
Selling your own beer onsite via the front door (i.e., patrons coming in to the taproom or restaurant) would be far more profitable than selling your own beer via the back door (i.e., via distribution to area restaurants, pubs, and package stores).
Do not underestimate how much time you'll need to do non-brewing activities (e.g., cleaning kegs, tending bar, cultivating relationships and sales). Go with at least 7 bbl, minimum. Every start up brewery in my local area that started with a 7 bbl system quickly "outgrew" it within 6-12 months and had to make plans for extra capacity, extra brew days (or adding a second brewer), and bigger brew systems.
Running a kitchen is very, very expensive and carries a high operating cost. Lots of headaches. If you have no experience running a restaurant or being a cook, consider being just a brewery with a taproom and have food catered in to guests. Your start-up costs will be far less.
Have as few partners as possible--easier to get decisions made and less splitting of profits. You will have to carry more of the risk, though. Don't quit your job. It always takes twice as long to get the place up and running, permits and licences obtained, etc. A lot of work needs to be done, but could be done after-hours, on days off, or on weekends. Even the brewing could happen after hours--you already do that. Having the job-security and income would help a lot and provide peace of mind until you are able to open your doors and assess the success or failure of your enterprise. Once you become a going concern with a proven revenue stream, it becomes a lot easier to expand.
One more thing, your friends who are encouraging you to open a place may also be great sources of free labor (in exchange for a few free pints later on).