With Battlefield 2 I go to the Battlefield 2 shortcut where you launch Battlefield 2
Right click on it, go to properties
Go to the Target path and change +fullscreen 1 to +fullscreen 0.
This is BF2 so i'm not sure if it would still be the same for Civ IV. I'm sure there is an easier way (Possibly in the Video settings area?)