DQ doesn't exactly fall in the cartoony style, not as much as Pokemon does. The comparison doesn't make much sense.
I read "cartoony" as "non photorealistic".
Like whether something looks like Mario, Borderlands, RIME, Dragon Quest, Pokemon, or even just Plants vs. Zombies, the usual divide is between games that try to look like reality or a somewhat stylized reality and games that don't at all (non-photorealistic).
Now, I think as time goes on this distinction is less notable than it was since there a lot of core oriented games have adopted more stylized art as opposed to the days of yore where it was almost more of a "this is for kids/families", "this is not for kids/families" split.
Games like Borderlands for example sell incredibly well (10+ million copies) and look cartoonish despite appealing primarily to the 18-35 set and being a violent M rated shooter with blood and gore.
In this example, it would seem to largely be a way to ask "What is the best selling JRPG in the West that isn't Final Fantasy or Dark Souls." It would be true that on consoles there really aren't any that do well, and we have seen games that fit this description succeed on handhelds.
However, I think this is also a somewhat odd lens to look at this through. It's not taking into account other non-photorealistic games that do well for one, including RPGs like Borderlands and Diablo, or the actual context of what the art looks like, which obviously hugely varies in non-photorealistic. It also focuses heavily on the art instead of analyzing the other aspects of the product that are likely to impact the game like demographics, relevance of the brand in various regions, core gameplay, extended functionality, and value proposition.
So, I guess my thought process is that I feel Pokemon is the right answer to that question, and art is certainly part of what affects Dragon Quest sales, but I don't think that's actually the logical way to keystone an argument of where the game should go regardless of which platform you think that should be.