I really don't see what major changes have been done so far that are markedly inferior to what's in the books.
Sure, Barristan Selmy was awesome and I hate to lose him, but it's not like his death ruined Dany's storyline, or altered it in any way.
Moreover, the Sansa thing (the other major change) is much more interesting than what is featured in the books.
What would Sansa do in the Vale this season?
Do we really want Stannis to ride south and get stuck in a snow storm, then trade misfortunes with the Boltons until next season?
I think having an existing Stark take over the northern rebellion stuff is much better than some character we have to learn about brand new.
There comes a point in every TV series when new characters become unwelcome. It doesn't matter how well they're written or acted. They need to be disposable, and the characters left out of the books aren't disposable, so they're not in the show.