Yep, bell tower matching is fucked. Killed close to 10 invaders/defenders in the past hour and not one of them could take more than three hits from my sword. They weren't doing much damage, either. No way these guys are near my SL/SH.
It's possible that in Bell Tower if you are in the covenant you just invade randomly whoever walks in there. I did get invaded by some people in PvP that seemed like they were a lot stronger than me.