I don't know how much banjos differ from guitars...I've tinkered with them, the principles are pretty much the same, albeit of course tuning differences and soundbox construction and such like. A cheap banjo will not necessarily lead you on to greater heights of playing, but if you aren't prone to playing, it at least will represent a cheaper investment. I've known plenty of folks who get motivated to play guitar, and then get convinced that they need to buy a Martin, and find out, wow, chords are going to take a few months of good work to get down right, and then they become an owner of a nicer guitar that many excellent guitarists don't even own.
A cheap banjo is guaranteed to not be sufficient for most folks when they approach the advanced or even intermediate stage, but it just depends on how certain you are that you'll hit those stages...I still have my beginner stage guitars, an old used Yamaha superstrat from the 80s, and I love that thing, crappy neck and pickups notwithstanding.
Put it this way, if you can make a Rogue banjo sound as good as it can, then you'll be then at the level you can make a more expensive banjo sound great, and you'll probably be happy to spend the money. But if you jump into name brand banjos, you're statistically likely to be the equivalent of that guy that bought the stainless conical fermentor cause he was told it was the best way and decided after his first batch that homebrewing wasn't for him.
Good luck regardless!
I agree with all you say. Holding a Martin in your lap won't make you play like Doc Watson. But, as you get better and better a "cheaper" instrument will hit its limits and you can't get out of a rouge what you can out of a D-18.
Not starting too cheap is important also because at first you will want to emulate what you hear and know and what you hear and know isn't played on cheap junk. The first guitar I had was nothing more than a piece of plywood with a fence post nailed to it. It was an Oscar Schmidt which is an old Indian word for "one grade above Dog $^&t . I almost quit because of it. Luckily, I played a friend's better grade instrument and soon up graded which kept me with it. After about 30 years I finally own a Martin

, not because it makes me a better player but because...well I've always wanted one. Now that I have it I wish I had it (and could have afforded it) 30 years ago.
Go find a banjo player somewhere and take them with you to evaluate what you want to buy in your price range so they can make sure it doesn't have any nasty buzzing and the neck is straight. Ask anyone of them and I guarantee they will be glad to help you out. But most important is get something. Learn music.