it isn't though, unless you assume that libertarianism is based on the notion that life is or should be fair. Life is what it is, but a voluntary interaction does not suggest that all interactions are necessarily equally voluntary for both sides. Take a man that wants to start a business for example and needs say, $10,000 as a loan from the bank. That's a voluntary interaction because the sides can negotiate terms for the loan and each side can either accept or reject the offer.
Now take the same man only he's $10,000 in debt and he needs to pay it off immediately or he's going to be referred to a collections agency. He could find a different loan at a high interest rate, he could accept the consequences of defaulting, he could mortgage his home, etc. A lot of his decisions are motivated by an involuntary factor, a debt he has accrued but that doesn't mean his actions and his decisions are not voluntary.