Comparing RPG dungeon design to Lufia II isn't really fair. Lufia II is a puzzle game masquerading as an RPG. And let's be honest, how many people today want their super realistic RPGs to suddenly grind to a halt in order to turn all tiles red or whatever?
As for the overall question, as easy as it is to say that gamers today are babies or developers are lazy or just focused on meaningless content or whatever, I think it is reasonable to step back and question if there is a purpose behind all of this. Besides level design, what ELSE is different between modern RPGs and 90s-era ones? I'll speak primarily about JRPGs since I don't have as much experience with WRPGs.
1) 3D graphics means movement speed is slower and rooms are bigger. Link in his Pegasus boots or Terra in her sprint shoes can zoom across a map in a couple seconds. But in the 3D space, we have to make characters look and move in a realistic fashion. And in order to get enjoyable movement with a workable camera, we need larger spaces to play in. So all else being equal, an equivalently complex dungeon in 3D space is going to take longer to work through than a 2D (or a crude 3D with a top-down camera) one. Which makes backtracking from a wrong direction more tedious.
2) Full 3D camera can make navigation less intuitive. With a top-down camera, you always know which way is North. It's easier to orient yourself through multiple turns and have a decent idea of where you are in the overall space. But with a camera that follows you, it's naturally easier to get disoriented and lost. It's harder to tell you've gone 100 ft in 3D space than it is to go 5 tiles. It's harder to remember which way is the exit. Sure, you can have tools like a map and compass, but then you have accusations that you're just following a map rather than navigating yourself.
3) Combat is more complex. Back in the day, random encounters was mostly about attrition and resource management. You'd use the same basic attacks over and over, with the hope that you don't run out of MP or potions before hitting your save spot. But people prefer more complex combat these days. And people dislike combat happening every three steps like the bad old days. So attrition is replaced by fewer but more challenging enemies. And in order to have more complex combat, it needs to take longer. Or action instead of turn based. But action-based combat that is over in 2 seconds is also boring, so it also needs to take longer. So you are taking longer in each individual combat experience. Which can hurt your short term memory of what the dungeon layout is. Oh, and if it is action based? Again, you are focused so much on combat that you lose all sense of direction. So it's possible to forget what door you came in...
4) You can save anywhere. Even if you can't, quick resume means you can quit anytime. Back in the day, unless you kept your console running all night, you couldn't. So developers KNEW you were going to tackle that complex dungeon in one sitting. But now? You get called away 20 minutes in and can't get back to it for a day or two. And now you forgot where you were, what you explored, and where you need to go to. That's a frustrating experience. And designers have to keep that possibility in mind when designing their games. I don't think it's a surprise that Metroidvanias, a genre that lives and dies by level design, has almost universally kept designated save rooms.
I'm not saying all games should have hallway design. I find it rather boring too. But I can understand where they are coming from. Of course, not all games are full 3D, so the first two caveats don't always apply. And not all games have complex combat. But there are enough hassles associated with complex dungeons that it's not surprising that many designers would rather cut them out than try to work them in somehow.
