If you want to learn faster without wanting to speak/write it but just read it, maybe try looking into a few textbooks like I did back in high school to learn a little quicker.
Pedagogical settings essentially have a rigid structure that won't really prepare you fast enough if you just, say, want to learn how to read a novel or play a game within a few years. You'll miss out on interaction and group activities which are generally used for speaking exercises, and feedback from teachers who would know more about idiomatic phrases or different readings of a character/lexical item if you decide to go the self-learning route, though. So, it's really up to you as to what you want to do.
Learning Katakana and Hiragana just takes two weeks if you consistently work at it, but you'll probably know a few characters here and there prior to starting because of constant exposure. It's the kanji that really takes time, and I usually subscribe to the belief of knowing !~2000 kanji would probably be beneficial before attempting a visual novel or something with a lot of text like an RPG.
It's not really hard; it just takes lots of practice. Especially since you're venturing into a terrain that doesn't use the Latin Alphabet. Generally from my linguistics studies, it's much easier to learn Dutch or French for native English speakers because: a) they learn something using the Latin Alphabet, b) Dutch is very close to English (but Frisian is probably closer), and c) French lent a ton of words to English back during the Norman Conquest of 1066. But no language is "impossible". It takes practice.
...get me talking about linguistics and I just ramble on and on and on...
