One problem could be the tendancy for bateria to form biofilms. These are aggregations of cells that adhere to surfaces and protect themselves with an extracellular slime layer that is resistant to simple rinsing or even the milder cleaners/sanitizers. Over time, these could become a source of off flavors or contamination. Biofilms won't necessarily for after the first skipped cleaning, or even the tenth, but eventually......