This is my view too. They have too many enemies that are only able to be taken down with specific elements. The shields on the abyss mages (opposing element only), the big shielded hilichurls (omnidirectional blocking somehow, pyro necessary), Fatui (opposing element only, very strict to a single element per fatui), large ice shield hilichurl (pyro again), Geo/Cryo infused giant hilichurl (Zhongli or claymores/pyro again). It's just too much. If the team swapping mechanic didn't take 30 seconds, it'd be better, but it's just annoying to run around the world with certain teamcomps because you have to stop, load menu, switch team, wait for team to load, exit menu, then kill that enemy, and go back to your other team. It's easier to run around with a boring comp with all bases covered, even if you don't like the way it plays, just for convenience. This is what does the biggest damage to using Anemo/Geo characters in my own opinion.
I'd love the ability to hot swap teams, but with like a 20 second cooldown. That'd be much better since you could be running around the world, and say "oh shit, it's a wood shield hilichurl, let me pop into my Pyro team", then fight them, after the cooldown wears off, just pop back to your original team and off you go. Kinda like the way we swap characters just with a longer cooldown. I think it would promote a lot of different team comps outside of Abyss.