That's funny because i was going to say the opposite. Comfort should be the #1 priority because your phone is with you at all times, so you need something that is not inconvenient for you to hold or put in your pocket. You can always carry a battery in your backpack/purse/whatever to plug it into.
Having to carry around a backpack to hold a portable charger seems a lot more inconvenient for me than a larger phone.