Are you using the spire shortcut?
You should be able to get to the Ghosts in 10 to 15 minutes, including the eventual deaths getting 4 players up there.
Now, that you mentioned it. I know of 2 tricks:
1. Using a Grenade and Hammer to get onto the 45° pillar and walk to the top of the tower. (Might want to clear the area 1st)
2. When the fight on the top begins, kill 5 enemies and then jump down from the tower. It will deload every enemy and the doors will open early.
Those are probably some very useful tips for this and for the LASO achievement.