At some point, you will reach a point of no return. However, I agree with majorvices in that you should wait until you package the batch to move it off of the primary yeast culture. If you are like most brewers, you will more than likely package the batch before it reaches the point of no return.