Plus if they are planning on using Avengers 3 as a vehicle to introduce/set up a New Avengers team, then it's going to be packed enough as it is and a lot of the current team members would likely only appear for supporting/cameo parts, especially if they would play a role in the larger universe cross-over event (let's just call it INFINITY for now).
The rumours of Cap forming a new team at the end of Age of Ultron makes sense if it's going to work this way. Maybe they get Evans back for both of the new movies and he leads the New Avengers team with Quicksilver, Scarlet Witch, Panther, Captain Marvel, Ant-Man/Wasp and Doctor Strange. Then for Thanos, they need to get the core team back together along with the New Avengers, The Guardians of the Galaxy, and The Defenders. It'd be massive.