There's almost 65 people on the list...what I've done was add 10 random GTs to my list so that maybe we can be more inclusive on any future races. MS really needs to change the 100 Friends limit...but in the meantime I think that as each of us have different friends we can all be interconnected that way. Ie. I have Truelize on my list and invite him, whereas you don't have him. At the same time I don't have Che on my list but you do, so you can invite him.
(If MS implements a clan feature like I hope they do, then we wouldn't have to worry about this because the clan list could be separate from the friends list.)