That really depends. If you have any invis threats on the enemy team you probably won't ever have the luxury of getting rid of it.
Typically you don't want to replace pink wards with a half-finished item (spectre's cowl, warden's mail, needlessly large rod) but only with a complete item. If you already have a pink ward out in a good spot (say baron or a good spot in the enemy jungle), you can replace the pink ward slot with good conscience. If you don't have a pink ward out then try to place one in a good spot before you replace it. If you're just gearing up for the final fight in the enemy base then whatever, get rid of it.
You really want to get elixirs before you get rid of pink wards though.
I don't have any ward hotspot maps, but you want to ward objectives that are about to be contended as well as covering your flanks for when you are about to contest an objective, for example jungle entrances near mid lane/baron/dragon. Laning wards are kind of obvious.