I think it's all about how much you enjoy the core "game" of it. The objectives can be pretty simple (rescue this guy, kill these guys, find this thing, etc.) but what makes it fun for me is just doing it however I want to.
I totally know where you're coming from though, I just could not get into Spelunky for the life of me and it was talked about NON-STOP on podcasts I listened too, especially Idle Thumbs. Part of that could be just having the game overhyped for me, when I hear too many great stories other people had playing the game, I find that my time with the game is rarely if ever that exciting. Games I know I'm going to enjoy I usually try to avoid getting too many impressions or reading too much praise of them beforehand, I skipped like half of the last Bombcast because of spoilers and just not wanting to know about certain missions or ways they completed them.