1/3 to 1/2 bottle should be fine for 5 gallons. If bottling, check the sugar amount you're adding via syrup and adjust the priming addition accordingly. They do make sugar-free versions, as well, but I tend not to use those in my beers if making a dessert type of beer or fruit-flavored beer.
Generally speaking, the advice to "start low and go slow" with flavor additions and spices is wise. Every beer is different, but I'd imagine that a milk stout could possibly handle up to a full bottle of the vanilla syrup (for a 5 gallon keg), especially if it is excessively roasty for your taste--the vanilla syrup will help mellow that roast a lot. But, I would start with 1/3 bottle and see how you like it first.