By Luke's time, force users were pretty much extinct and considered relics of a bygone age who practiced some weird religion. No one knew the Emperor was a force user except for Vader, Yoda, and later Luke. Grand Moff Tarkin even said that Vader was the last living Force user in the original Star Wars because everyone else thought Obi-Wan Kenobi was dead. Luke was the last Jedi, and he only whipped out his saber to fight Vader and Jabba the Hutt on the barge.
The reason it seems different to us, is because we're following Luke's journey as a Jedi in the OT. Everyone else in that universe didn't see it from his vantage point. The only ones who experienced it first hand were Han and Leia because they were with him for most of it, and witnessed his exploits first hand (also Leia herself being force-sensitive). So for people outside of Luke's circle (i.e. people who were born years after RotJ), the Jedi, the dark side, the Force, just seems like fairy tales and urban legends.
In short, a person living in the Prequel era might see a Jedi in passing. A person living in the OT era hasn't seen a Jedi in decades. A person living in the new trilogy era has never seen a living Jedi, and their exploits were either tarnished by the Empire, or largely considered nothing more than tall tales.
So yeah, it makes sense.