The buffers should be non-hazardous. The MSDS (Material Safety Data Sheets) can usually be found on the manufacturer's website. From one site pH 4 is listed:
http://www.hannainst.com/sds/SDS_HI%207004_2013-04-01.pdf SECTION 2:
HAZARD IDENTIFICATION
Non-hazardous product as specified in Directives 67/548/EEC and 1999/45/EC.
Non-hazardous product as specified in OSHA Regulation 29 CFR 1910.1200.
Non-hazardous product as specified in Canadian Regulation SOR/88-66.
Non-hazardous product as specified in Regulation (EC) 1272/2008.
SECTION 4:
FIRST AID MEASURES
After Inhalation: Remove to fresh air. Call a physician if breathing becomes difficult.
After Skin Contact: Wash affected area with water and soap.
After Eye Contact: Rinse out with plenty of water for at least 15 minutes. If pain persists, summon medical advice.
After Swallowing: Wash out mouth with plenty of water, provided person is conscious. Obtain medical attention if feeling unwell.
The statements above are general statements. To put it in perspective all pH probes are calibrated in buffers and in the food industry placed in food after calibration. Ultimately, if a child consumed some buffer then it would be recommended to download the MSDS and contact a medical professional.
Use the link below to see what typical chemicals are used in making a pH buffer. The chemicals listed are what are purchased to validate a pH buffer (NIST traceable). For example pH 4 has potassium hydrogen phthalate.
http://www.nist.gov/srm/upload/SP260-53.PDF