I absolutely love the game but I too have zero interest in building stuff. I cant get the motivation to build things and when I do start I just give up. The reason why I love it is the exploration, the fact that you can have micro adventures, hoarding valuable materials and creating cool machines. Thats the beauty of the game, there is something for everyone. The fact that the game inherently has a high barrier to entry understandably puts people off.
There is nothing better than putting on some music, a podcast or tv show, equipping all your mining items and exploring the caves for a couple of hours. I find it borderline meditative.
Exploration is what does it for me as well...I have no time or inclination to create anything that remotely compares to what other have already created in minecraft. I just enjoy creating a simple tower or seeing how far down I can dig on the PC version in Hardcore mode. I bought the 360 version but the draw distance and limited size of the worlds kind of irks me.