I guess the thing I'm fearing the most is that the Adventure content will end up being too MMO-lite. More KILL TEN GNOMES than something that goes along with the open, evolving feel of playing Survival as we know it.
Also, when I imagine something like the goblin invasions from Terraria happening in Minecraft, I imagine them being more annoying than epic. Terraria is simplistic, but the battling feels very tight. In Minecraft, everything is far too loose and unpolished for something like that to be fun.
Either way, I'd rather have Minecraft fail with an optional mode than to stop moving forward. I'm very excited to see the Mojang take on adventuring even if I go back to standard survival.