Just a thought: Try brewing the full 5 gallon batch as normal next time, letting it ferment in the primary, then siphoning off 1 gallon into a glass jug. Add the ginseng tea to the 1 gallon test jug. This is what i do with my more experimental brews because 1) if it sucks i don't ruin the other 4 gallons, 2) it takes a lot less of the experimental ingredient to make a difference.
I have thought of using ginseng myself. As was mentioned, fermentation changes a lot and i wonder if it would destroy the "active ingredient" of ginseng so you wouldn't notice any effect on memory and energy. Not sure, just wondering. To be sure, you could add the tea to the secondary.
Like you said, the tea is almost undrinkable sometimes. I wonder if enough hops would counterbalance the flavor of the ginseng?