I don't think it's a matter of being difficult. It's manual labor but it's not *hard*, per se. It's super monotonous, virtually non stop (you get ten minutes of personal time but let's be serious), and the pay is simply not worth it for the conditions you go through and the stress you put on your body. You can toss packages all day with a blank mind, but it is draining on the soul, and mentally disengaging.
It's one of those jobs that humans really don't need to do, and indeed, package handlers probably won't exist at all, anywhere, in fifteen years. You don't accrue any skills, you are basically a cog in the machine, the compensation isn't livable, and you'll pick up calluses and bruises and cuts and scrapes and back pain and joint pain along the way. Especially if you do it for a while looking to get the benefits.
You're better off applying for a supervisor position. They seem to give those to anyone in school or with a degree. You'll still be tossing packages every now and then, but the pay is better and you actually do other stuff.