The flavor components that make beer taste fresh are antioxidants. Any oxygen that gets into the beer, that is not scavenged by the yeast, will stale the beer. This is why brewers strive to protect the beer from oxygen. The fact is you did do damage, how much is open for debate.
Part of what brewers perceive as staling in finished beer is that it lacks a viable carbon source other than ethanol. When finished beer that still contains viable yeast cells is exposed to O
2, the cells consume the O
2 and under go a diauxic shift where they start to consume ethanol as their carbon source aerobically, reducing ethanol back into its precursors. That does not produce the flavors we associate with truly oxidized beer, which is the result of filtered beer being exposed to O
2 during packaging. The reason why brewers still bottle-condition beer is that the addition of priming sugar gives the yeast cells something other than ethanol to consume after they take up the O
2. The modern process involves filtering and then adding back a small amount of yeast with the priming sugar.
Wort is not finished beer. It is rich with sources of carbon. The main threat that arises from waiting to pitch a yeast culture in wort that has been aerated comes not from staling, but from microbiological competitors, especially aerobic microbiological competitors, mainly wild yeast. Remember, a brewery is sanitized, not sterilized. There is always a small amount of house microflora in every fermentation. That is big part of why breweries tend to have a house flavor.