I'm one of those people. I hated Other M with every fiber of my being. I think if I took the game as a whole and considered all of the aspects of the game logically, it's probably not as bad as I think it is.
But that's not how the brain and memory works. Those terrible, character-assassinating moments in that game are literally all I can remember about it. It's burned into my head. And as someone who grew up on Metroid to the point of it probably being my favorite video game series, it genuinely put a stain on Samus's character that was tough to wash out.
I mean, there are plenty of other dogshit aspects of that game, like being forced to use pointer controls to fire missiles, and not being able to use just a normal controller in general. But that kind of stuff can be washed out by otherwise great experiences. It's why Metroid Prime 1 is considered a masterpiece, even though there's a pretty long slog near the end in collecting the remaining Chico Artifacts. Same thing with Wind Waker and the Triforce Shards. Those parts in those games could definitely be improved upon, but it almost doesn't matter, because what sticks in the front of your memory at the end of the day is the lasting impression you'll have of that game.
Other M probably had some good parts, but all I can remember are the parts that make me cringe, and actively destroy my perception of an otherwise stoic, awesome main character. It'd be like if your dad was an awesome guy and great father your whole life, but then one day you came home and walked in on him fucking the dog.