I'm fine with Spider-Man being with Sony and being purely solo because he has enough going on with just him and his villains. What makes him interesting is his regular life versus his duty to the city. Him going on missions with the Avengers kinda doesn't fit into that other than "oh he's funny so he'll be funny with the Avengers now." Besides it makes all his threats seem really inconsequential when the Fantastic Four, the Avengers, and all these other heroes are right there in New York as well.
The X-Men universe is complex enough on its own without the Avengers getting involved, and frankly are better off being separate. Would the Avengers really let a Days of Future Past story happen? What, Thor and Hulk can't take down like 20 of them at once?
Fantastic Four I'd want with Marvel if purely to let the Avengers fight Doom or Galactus or whatever, but otherwise, I don't give a shit about the Fantastic Four as characters.