You are correct, the yeast is in the liquid in the main pack. The interior pack has yeast nutrient and a bit of food for the yeast. If you're making a starter, you don't have to wait for it to swell first, but I would definitely smack it to get the goodness from that interior pack into your starter. If you're concerned about the yeast being viable (i.e. shipped in hot weather, really old...) then wait for it to swell for peace of mind. I typically get my starter boiled and cooled, then pull the pack out of the fridge, smack it, and pitch it into the starter immediately.