Yeah, I think it's way too long. I mean at that point we're looking at literally a decade's worth of buildup, on top of the issues you mention (I think genre fatigue and actor issues are their two biggest hurdles).That would suck. Dragging on the build up for too long. And by the time Avengers 4 rolls around, much of the current cast will have left due to contracts being up and a fourth movie with them being mega expensive because of their salaries.
We have spent this much time building to the Infinity Gauntlet. Dragging it on another 3 or so years after Avengers 3 would be bad.
Just spend Phase 3 fleshing everything out in the other films, then make Avengers 3 the big get together to stop Thanos. Only thing I can see them doing to be bigger than that is making Avengers 3 the holiday 2018 movie and then the may 2019 film being simply called "The Infinity Gauntlet" and throwing everyone in it.
It's not like they'll run out of ideas for Avengers 4 either. Sure, there's no villain quite like Thanos that they can use that has the same impact (especially after Ultron getting his own film and Doom being restricted by rights), but there's always adaptations of Disassembled/Civil War and perhaps even the Thunderbolts they can use.