I think it did hurt it a bit. Four years may not seem like a lot, but it's enough time for the movie landscape and tastes to change. I think that if they had really wanted to strike while the iron was hot, they would have had it out 2-3 years after TDK.
I'm not sure about this line of reasoning. The reason they waited 4 years was for Nolan to do Inception which ended up being a huge hit for them. If they tried to get it out in 2-3 years, then they probably wouldn't have gotten Nolan to write/direct it.