No, you can't remove them, at least with a rare mod it is just replaced and you lose the original. They are consumables that you acquire in large quantities.
And I can answer why it isn't clear and why it's being downplayed at the same time I think...
a) Legendary mods aren't like, something I can even make the choice to have yet
b) Most people don't yet have full loadouts they like yet that they are sure they want to mess around with modding
c) Rare mods at least don't have that much impact at all (all the buffs are slight), except changing element which is self-explanatory
so it's just not a big part of the game for me yet. You say "especially for the long-term" like we're supposed to fully grasp the long-term now, the game legit came out yesterday