I still like Race for the Galaxy best with the base game and the first expansion (The Gathering Storm). TGS doesn't add much in the way of rules or cards (there's only like 20?), but the cards do round out the game's strategy space and the goals (e.g., 3 VP for first to have worlds of all four colors, 5 VP for largest military at end of game) are great.
The rest of the first arc adds additional rules (PvP combat, Prestige tokens) that make the game more difficult to manage and more swingy. Unless you really need more variety than TGS provides, I don't think these are worthwhile additions.
The second arc consists of a single expansion, Alien Artifacts, which has a good, balanced card selection (larger than TGS) but has a dull sequential Orb exploration mini-game added on in the place of goals. AA is fine for what it is without the Orb game, but the lack of goals is unfortunate (and the RftG on the whole is not designed to allow mixing-and-matching between arcs).
The third arc also consists of a single expansion, Xeno Invasion, which also has a reasonable card selection (again, larger than TGS) with focuses on stronger military and faster consumption. The new Invasion mini-game requires fending off attacking NPCs every round, which adds a lot of pressure and greatly changes the feel of RftG. I'm not entirely sure how I feel about XI, but it's definitely different with the Invasion option, and without the Invasion option it doesn't feel as interesting as TGS. I agree with Musan that the meld on all explores does really help, though.
In summary, TGS is great (and honestly ought to come in the box with the base game). Don't get the other expansions until/unless you get bored with base + TGS.