I don't think the learning part of Japanese is particularly hard if the goal is to consume media. I've been studying on and off for a year and a half now, with a couple of intense periods here and there, and I can handle a lot of media if I have a dictionary by my side. I would say keeping track of a real life conversation is much harder for me because of how much the topic of a conversation can change, so it's easy to get lost.
On the other hand, I think production is really difficult . How some things "work" but don't really sound natural for a given situation, or how some patterns or conjugations simply don't apply to certain kinds of words and ways of speaking, even if they kind of make sense. Like 可愛そう (かわいそう
doesn't work to say "Looks cute!" even if you think it would because it actually it means 可哀想 (かわいそう
. I used this one as an example in a post I wrote a little while back, because it appears correct usage, but isn't for a couple reasons. I'm surprised no one picked me up on it. The lesson being that rules are okay to learn, but often they don't work for a given situation even if they appear to. Rules are just their to help you process how the language is being used naturally. Therefore the best way to learn the correct usage of rules is through exposure and that takes a long time and a lot of skill because first you need to know all the rules, and then you need to be able to process and reproduce to say the things you want to say.
I suppose this is common for any language though. Once you delve down the rabbit hole and actually start trying to express things you want to say in an interesting and natural way instead of stuff like "寒いよ!" or "新宿が好き!" then I think Japanese is hella difficult. This the opinion of someone who isn't very good at production though. I have also never learned a second language, so learning Japanese is a completely new experience for me.
Would be interested to hear other peoples opinions, especially the J-masters out there. We don't need to keep talking about how 上 can be じょう or うえ like it's a crazy mystery, come on people.