That's why chalk (calcium carbonate), if used at all when building your water profile, should be added directly to the mash.
Otherwise it just remains a precipitate in the bottom of the hot liquor tank (BTDT).
Boiling causes calcium carbonate to drop out as precipitate--it is an effective method of pre-treating one's indigenous brewing water if the bicarbonate levels are too high. There's much more information about bicarbonate levels, pH and alkalinity throughout this forum--much of it contributed by Martin and Kai.