Not a chemist here. But I believe it has more to do with its not oxygen.
CO2 is the primary gas produced by fermentation. However, it is of course far from the only gas emitted. There's also various sulfur compounds, aromatic esters, alcohols, water vapor, etc. You are correct -- none of these are oxygen. The only way to produce oxygen that I know of would be via electrolysis by running an electrical current through the water like a battery, and I seriously doubt any brewer is doing that! Nevermind the flammable hydrogen gas that would be produced along with it!