IMO, any more than 4-5 different types of grain (in a normal strength beer) just gives a muddled, unimpressive end flavor.
On the other hand, I know stout usually requires the Big Three (Choc. Malt, Roasted Barley, Crystal), and I LOVE Special B in my RIS, so why not in a medium-strength stout?
I'd pick ONE of the other malts to use (Special B or otherwise) and see what you think. If you don't like it, try another next time.