1) Most of the game's mechanics are at the very least poorly thought out (cross rush combos) and at the worst either useless or detrimental to the game (pandora, gems)
2) The game had significant bugs - various characters had infinites, Rolento had a ZA WARUDO game freezing glitch in one patch, Megaman could fly into space...
3) The game's balance is badly off. Much of the cast lacks a reliable anti-air, either normal or special. The top tier characters typically have amazing jump-in attacks that exploit this. It would take far too long (and admittedly far more experience with the cast than I have) to list all the balance problems. This is the main issue!
4) Wonky hitboxes. This ties into 3. Cross-ups are AMAZING.
5) The game encourages safe pressure strings. In SF4, getting hit with a standing light punch only leads into small amounts of damage and sometimes a knockdown - in SFxT it can be a third of one of your character's life. Result? Everyone blocks. All of the time. There's very little between BS mixup and safe pressure.
5) Due to the game's 2v2 nature combined with SF4 damage output (certain characters notwithstanding) you run into timeout after timeout.
6) The game is visually ugly and the music score is awful!
Also, PR disasters and DLC.