I don't remember the store ever showing that as compatible... are you in the US?
Calling is always better than email, if possible. The confusion over 'Rivals' is likely because most customer service people don't have comprehensive knowledge of the PSP library so think you're talking about the newer Rivals.
Call, and you should be able to get it sorted out. If not, just ask for a manager.
Don't bring up past calls, don't describe the situation too much... just say it doesn't work on your Vita. The more you talk about, the more you can confuse the other person -- keep it simple for best results with customer service. If it were me, I'd seriously limit it to "Hi, I bought this game, Need for Speed Rivals Underground by mistake -- I thought it said it would work on my vita, but it doesn't. Can I please get a refund?"
My guess is you should easily be able to get a 'we'll do this for you one time' speech, which you can ignore, they just say that to make it clear you shouldn't abuse it, they'd do it again if they have to.