Laziness and brewing don't mix. Clean the keg! Sanitation is key. It takes me about ten minutes to clean a keg. Just devote a few minutes to clean it and then rack your new beer, it's not like pouring a different beer into the same uncleaned glass, the beer will sit in there for x amount of days/weeks while you drink it, I'd worry about imparting undersirable flavors.
Also, I brewed a 10 gallon batch of beer that had some fusels or something going on. No headache but hot alcohol taste. The first 5 gallons it was very noticeable, now I"m drinking the second 5 gal, and it's not even noticeable.