You guys are probably right. I've never heard of phosphate poisoning, but alcohol poisoning is common and lots of people die from that everyone year. The person was basically telling me "don't put poison in your poison."
Phosphate is a normal electrolyte present in your body. A typical IV dose of sodium phosphate we would give to raise someone's phosphate levels if their PO4 is running low would contain somewhere in the ballpark of 500-700mg of phosphate ion. I typically add less than this to an entire 3-gallon batch (10% Phosphoric acid contains 100mg of phosphate per mL). I don't think there is anything to be concerned about with the levels we are dealing with in the brewery.