I read the whole thread (including OP obviously, yes, I had a bit of time on my hands).
First off, OP, it's nice that you were able to broaden your horizon and came to understand something you previously were confused/appalled by.
We should all aim to understand one another and not mock other viewpoints and preferences as long as those are not hurtful to others.
So, thumbs up for that. Also, Persona 3 is one of my all time favorite games, so it's cool that you had an amazing time with it.
Now, as for some of your questions:
I only played Persona 2: Innocent Sin and a little of Persona 2: Eternal Punishment when it comes to the older Personas, but let me tell you that they are vastly different than P3 and P4. There is no day to day management aspect, so they're quite focused on dungeons (whose design I did not find particularily great). Couple that with a high encounter rate and it can get quite annoying. I think mechanically, these games are really clunky. I personally didn't care much for the cast or the story either, but many people actually like these games casts so if you can put up with the gameplay for the characters, then go for it. I say the games are definitely worth a try. Just don't expect something akin to P3 just because it says "Persona" on the box.
As for P4/Golden, Golden added a lot of things, some smaller like additional costumes, and some bigger like additional scenarios and a redone fusion system.
I haven't played P4 Vanilla personally myself,but watched a let's play of it and it is definitely worth playing if you don't have access to a Vita. Persona 4's great qualities, like the characters, the addictive gameplay loop and the engaging SLinks are all present in the Vanilla version, so P4 PS2 is still a perfectly fine choice.
As for other SMT games, I personally have played Nocturne, DDS1 and DDS2. They are great games, but tonally very different from Persona. No social link aspect or anything, more focus on gameplay with elaborate dungeons. Combat in Nocturne and DDS1/2 is similar to Persona, but with a few changes that make it more engaging and challenging. The atmosphere of these games is also the exact opposite of cute. It's a tense, oppressing atmosphere. you get thrown into a world that's fucked before you can even do anything about it and now have to find a way to survive in that hostile world. Nocturne is very gameplay centered and the story bits are few and far in between, but what's there is intruiging. DDS on the other hand puts quite a lot of focus on its story, with lots of cutscenes and dialogue.
You have to see if those are for you. Definitely great games, had a lot of fun with them.
