I don't think you should be disappointed. His hands were bound, he's tired and has been malnourished for a good bit, and he was unarmored fighting a giant opponent in full armor. That's a lot of disadvantages, there.
His quote about there only being 3 swords in the kingdoms equal to his is stretching the truth a bit. He's a Lannister, afterall.
Didn't Ned almost beat him before one of Jamie's men intervened? I just think that no one had ever given Brienne a chance to prove herself. If Jamie was healthy, had his armor and it was a legitimate fight, he'd probably have won. You don't become a kingsguard and be the Kingslayer and challenge Ned who has a Valyrian steel sword unless you're confident you can win. Who knows though.
Margaery seems nice enough, but she's smart and is playing the game of thrones, either you win or you die. I don't think she has malice in her heart like Cersei does.