Yes, if you pretend that there is absolutely no context to it at all and that he hasn't been dogged with questions about his treatment of women in his work for years, it's completely harmless.
Back in reality, however, it just winds up undoing what was a really good answer to the question.
I think it's a pretty harmless joke at Helen Mirren's expense but as you say, bearing in mind all the accusations that have been levelled at him in the past it was a pretty stupid thing to say.