Don't worry man you are not alone, we are definitely a small minority as most people have no issues. I don't have the best internet, so I always thought it was "guess my connection isn't strong enough to play, bummer" but my friend who has an even worse connection (500 kbs down, 300 up) can play it without a hitch, so it's definitely some strange problem with Rockstars servers, but Rockstar has largely ignored this minority that still, a year later cannot play online.
I've done combinations of everything possible on my end, set Xbox in a DMZ, Set Xbox to Static IP, Port Forwarded, turned Upnp off, turned it on, turned all firewalls off, Open NAT, called the ISP to see if they could do anything on their end. Nothing ever works, at best I'll join a 30 player server for a few minutes and I get booted the first time I come in close proximity with any other player (that or it kicks the other player I get close to, the fuck is wrong with this matchmaking)
It's a shame cause I'd really love to have fun with my friends, but I can never join them...or anyone for that matter.
This thread has been the most helpful on the issue, so it's worth a look, might work for you, but it seems my problem is something that cannot be changed:
http://gtaforums.com/topic/692081-fixing-lag-issuesconnection-timeouts/