Elderflowers are toxic, but only mildly so (the berries much more so, and the green parts of the plant). It's a nerve toxin. The toxin is destroyed by heat (boiling water poured over them will do). I use them extensively.
The 'your father smells of Elderberries' as a perjorative has a historical basis. Elderberry Wine (which is very good but needs aging as a result of high tannin content) was once known as English Port. The berries were even used by Port Wine makers to boost the grapes in bad years, before the practice was prohibited. Since the various Port houses at the time were all British owned, the French criticized the English for debasing Port, and for making a substitute with Elderberries. Hence, we get to the French taunting the English in Monty Python and the Holy Grail.