After spending more time with it, there's definitely a couple of changes I can think of that would dramatically improve casual matchmaking. The most obvious ones - faster matching and more quickly refilling matches when players leave - are already in the works, but if I had to suggest other major changes:
1. Voting needs to come back.
In the event of hackers/cheaters/griefers, there really needs to be a way to votekick disruptive players. Additionally, an option to votescramble would be welcome if team sizes get uneven, as a way to solve player count disparities in cases where teams don't refill quickly enough.
2. Matched players should stick together through multiple maps.
There's no reason to force a re-match after every single round. You've already found a set of players that are interested in at least one of the same gamemodes as you, so it's pointless to refind them every single map. After a round, players should be dropped into a lobby where they can leave if they want to, teams have time to refill if players leave, and the next map can be voted on. This is how basically every other god damn FPS works, and it makes no sense why TF2 doesn't.
3. Incentivize players to stay through the whole match.
I'm not sure how this would work, honestly, other than an abandon penalty, but there should absolutely be a deterrent to ragequitting. TF2 is a team game (it's right in the name!) and five people ragequitting is a selfish move that fucks over your whole team - even if matchmaking worked faster at refilling slots, there will still be a lot of downtime which can cost a team a game and, more importantly, just be exceedingly unfun and frustrating for the teammates who are left.