You (or a 6yo kid) go into online mode, and suddenly this guy is dancing around the screen and you're really confused as you can't keep up and get frustrated and quit. That's not a good thing to happen.
The amount of people playing party mode dwarfs the amount of people playing competitively. That's true of all games. And yes, you cater to them first and foremost. Balance, things to learn, matchups, etc- those things matter, and are important in the long run because they trickle down, but they're knobs you can tune. The bigger building blocks you have to lean towards the general audience (assuming it's a mainline title, not a niche one) in order to hook them and keep them invested/playing and happy.