My idea is that it's because Earth is Asgard's pet planet. Asgard is a huge famous intergalactic society - even the Collector respects them - and most Infinity Fuckery is a direct result of Asgardian nonsense.
Odin left the Cube there, stuck the Dark Elf Red Goop nearby, and Loki brought the Mind Gem.
Earth is "that shithole where Odin's family keeps slumming it."
Dont forget the time gem. From what we know so far from doctor strange the time gem has been in the sorcerers' possession for a long time.