Hmm. Yesterday I found a solution to something that had me really concerned about Game Maker: Studio. I'll share it here in case any GMS users have the same problem:
If you are working with a lot of really big backgrounds (like I'm doing), you might have seen yourself in a situation where Game Maker can't create the texture pages because there's not enough RAM memory, thus causing the compile to fail.
You can try smaller texture pages (which will increase the ammount of texture page swapping and affect performance), or even freeing more RAM before compiling (which can be inconvinient, since compiling times are long and you might want to keep doing things in the meantime).
But if you keep adding backgrounds and other resources to your game, you'll unavoidably come to the point where you need more RAM memory.
Now, lets say that you can't upgrade your system's memory. Maybe because you don't have available slots in your motherboard. Or perhaps adding more RAM would be pointless because the architecture of your OS doesn't allow it (as it was my case, being a 32 bit Windows 7 user with 3 GB of RAM).
If you can't/don't want to buy a more powerfull machine, there's a solution to effectively compile your game and all your expensive graphic resources:
1st: After the compile fails and you get the out of memory window, close the game window and stop the debugger/player.
- 2nd: DO NOT CLEAR THE COMPILED ASSET CACHE. It's the first thing that the Game Maker Forum users will tell you to do -> "If compile fails, clear the cache, free ram, and try again". This might work sometimes, but it usually means that you're on the edge of not having enough memory to continue making your game.
- 3rd: Hit compile again. Since we haven't cleared the previously compiled assets, GM won't try to create again the troublesome texture page.
- 4th: If you keep getting "out of memory" errors (because there's more than one texture page that GM can't create), repeat 1, 2, and 3, until your game launches. I only get the out of memory error once (because of a certain amount of particularly expensive backgrounds that belong to the same texture group).
- 5th: Now, here comes the tricky part: finding the texture page that GM couldn't create. In my case, it was pretty easy since it was the last texture group I created. If it's not your case, you can check the temporary directory where Game Maker stores the created texture pages, or you can go through your entire game till you find the room with the failed backgrounds. You'll recognise them pretty easily: they'll look, glitchy, cut and/or smeared.
- 6th: Now, close your game, and in the GM editor, open one of the backgrounds that belong to the same texture group of the failed texture page, edit it (paint a transparent pixel for example) and save the changes.
- 7th: Compile again. Since you've changed this texture group, GM will try to compile its texture page/s again. But this time that will be the only thing that GM will compile, since the rest of your resources are already there from the last try.
Now, Game Maker should at last be able to compile that stupid texture page. If it doesn't, well, you'll have to free some RAM and try again. and if It still doesn't compile it... you're definitely trying to compile something that's too big for you system to handle. Try splitting it into 2 or 4 different backgrounds, then drawing them side by side.
This worked for me, and allows my 3GB of RAM system to compile 2048x2048 texture pages without having to close other programs (photoshop, skype, etc).
I hope it helps! (what I really hope is that you don't have the shitty PC I have).
EDIT: Before doing this, obviously... optimize your resources! Avoid big, repetitive backgrounds if you can make them smaller and draw them tiled or stretched. Crop your sprites and backgrounds to get rid of as much empty space as possible, even if that means having to split your picture in different parts. Most gradients cant be reduced to a 1 pixel wide background that can be stretched to fit the screen width. If all your animated sprite does is rotating or changing its colour, do it with code instead. I remember this article being specially useful >
http://www.yoyogames.com/tech_blog/30