A combination of stability, money, and time. Nintendo and Sony also both use the same system. Compile against a version of the console OS, ship that game, game can't adopt any new features. Games ignore newer features so MS doesn't have to go back and test all games to make sure they don't explode on newer dashboards.
One of the more invisible improvements that was revealed later by a Bungie interview? In TU2 Halo 3 actually has more RAM available for the game to use, because they figured out how to free some up. I thought that was neat, since it was an important update and probably hardly anyone would have noticed it.
The biggest improvement though was when MS finally gave devs twice the RAM in the devkit, so you no longer had to run debug stuff inside the system's 512MB, so later games actually have more overall RAM to use. Like how later PS3 games can use more RAM because the first PS3 OS reserved 128MB of RAM for itself. 128MB! on a 512MB console!