It's easy mode whether you like it or not. Having to do something faster is adding another layer of difficulty. Making a work of art is much easier if you have 20 years to do it instead of 5 minutes.
Now, why is it integral? Simple, the dungeons are designed around to be beaten in the 3 day time limit. Even if you're slow they even give you a "checkpoint" if you are too slow when you get the relevent dungeon item. You can essentially wind back time in the dungeons once you get the relevent arrow and return to exactly where you were. EG, stone Tower temple, you get the light arrow, you can instantly flip the dungeon over if you go back in time and essentially nothing has been lost other than fairies.
Majora's mask Is ALL about time management. "Should I go for the fairies now or try and finish up the quest for the postman's hat?" Having to choose between sidequests and dungeon exploration is an integral part to what makes Majora's mask enjoyable for many.