The extra bulk if you're using one of those battery cases like mophie makes, and having to carry around an external charger compared to a very slim extra battery. That said, I could live with using an external charger, but I'd just rather have a second battery.
I'd rather have a premium/well-built phone as well. Plastic is fine with me though, as long as it's not glossy and doesn't feel like crap. I've played around with so many other phones since getting my S4, and they feel so much better naked. I use a case to mask the gloss, but would prefer not to.
Also, I'd rather not have an all metal phone. They are too smooth and slippery. Matte and textured plastic is the best in my opinion.
----
And didn't HTC make an aluminum phone in the past that had a small plastic(?) section on the bottom/back that could be flipped open and the battery could be slid out? I think that could be a great way to have a metal/premium construction while keeping the removable battery and/or storage.