I don't think they're merely agents of controversy, but I do think they cause a lot more trouble than they are of use. It's nice and all that everyone has a different opinion over what a 4 vs a 5 vs a 6 means to them, but giving a highly reviewed game a 4 is going to create controversy whether you mean to or not.
It's a something you'll see again and again come up, some highly reviews game getting a worse score than average, and then people loosing their god damn minds over it. Most don't even bother to read the review, and see what the actual criticisms are, and it really hurts discussion.
In a perfect world, people would always read a review before commenting, and a review score would be a way of simply summing up their thoughts on the individual components of a game, similar to how a grade is just a summation of different problems and assignments. The problem is most people seem to argue over the score itself rather than the reasons over the score. I'm guilty of this myself, getting caught up in threads where a game earns a lot of 10's or whatever.
I simply think the discussions of games would be a lot better if there wasn't a number attached to a person's thoughts. That way, the discussion would have to be on the criticisms and praises of a game instead of the score attached to those.