I've never used chestnuts in a beer, but the latest issue of Brew Your Own magazine (March-April 2016) has a recipe for a chestnut beer in an article about craft brewing in Italy.
As far as BeerSmith goes, you can add new ingredients to BeerSmith's data files. I think I'd likely add chestnuts as a miscellaneous flavor ingredient, like coffee, rather than a malt. But I haven't done any research to support this.