The thing is, they weren't implying he was saying anything though. They were actually drawing attention to the fact that he was staying silent (Alyx: "You don't talk much, do you?"; Breen: "I can understand if you don't want to discuss this in front of your friends," etc.). If the other characters had acted like Gordon had actually been saying stuff (like the characters in most other games with silent protagonists do), I would have been okay with it, but when they expected him to respond, it just felt weird. Basically if your game is going to have a silent protagonist, make it clear that they're being a silent protagonist, don't just make them a protagonist that's being silent (if that makes sense).