The problem is that women are generally smaller and physically weaker than men. So if a man hits a woman, the injury to the women will be far in excess of the threat she posed to the man. The disparity would be the same, if we were talking about an 80 year old man and a 20 year old. The better course is to push her away, leave and/or call the police.
You do realize your basic argument revolves around the fact that because a man his bigger and stronger he deserves to be unemployed and live in poverty while a women, because she is small and weak, should be able to find employment and move on with her life and get a job?
I honestly don't know how you dont see that as totally unfair and wrong