It's not "hard" it just becomes uncomfortable and I don't think health or fitness has anything to do with it; though it certainly
could be a contributing factor for some. Even in my prime with no (apparent) medical issues playing a handheld system became uncomfortable after a period of time.
In fact if you look at the picture in
post #5 I can point to where a problem lies for my hands. Look at where the ring finger is positioned on the back of the system to help support it. The weight of the system combined with button presses causes that last joint to bend inwards with a similar thing happening with my pinky. And being focused on the game I won't notice because it doesn't immediately feel wrong. It's not a problem for a small amount of time but after a bit it will ache and I need to readjust, stretch fingers, etc. I've always had this issue with handhelds and even the flat controllers from back in the earlier days of gaming.