Seconded. My favorite place in the state to vacation at.
It's cheap, has a ton to offer for outdoors and water activities, and still the "forgotten coast."
OP, what time of the year are you looking to vacation at? Since you're planning soon, I'm guessing you're thinking Winter/Spring.
Not to mention that it's peculiarly overlooked. Most people love to crowd Destin and Panama City, and there's no denying how beautiful those beaches are. But Port St. Joe is just as beautiful and way more low key.
Another place OP may want to look at is
Dog Island. It's inaccessible by land, requiring a ferry in from Carrabelle. There are private charters and daily ferries to take you out there.
This place is for totally turning off. The one "hotel" on the island is more like a duplex and functions more as a place to stash your stuff, take naps and breaks from the sun, make drinks. It's closer to camping. There are no stores, so what you bring is what you have. The water isn't potable, so bring that. Cell phone reception was, in my case, nonexistent.
But if you like seclusion, waking up to families of dolphins in swimming distance and utter relaxation, this is the place.