American home and craft brewers have developed distinct styles under the two names, but you're opening a can of worms if you want to definitively pin down the distinction. How many angels on the head of a pin and such. For competition purposes, see the style guidelines.
Historically, you're right, stout and porter were just the same thing at different strengths. They were usually parti-gyled (literally, "split batch,") which in traditional British brewing practice means a single strong wort is brewed, split into multiple fermenters, and each fermenter diluted with water before pitching to different gravities to produce different strength beers with the same flavor profile. (If this affects color, the weaker beer may be colored with caramel to correct this.) As gravities of all British beers dropped drastically during the first half of the 20th century, brands designated as stouts ended up at the gravity where porters had once been, there was no room to go any weaker than these "stouts" under the established excise tax schedule, and the name porter disappeared.
So the difference depends on when and where you are talking about.