There are some pretty obvious differences here, though, such that someone who wants to say that men have a duty to be careful not to make women uncomfortable isn't necessarily committed to saying that black people have the same duty re: white people. Let's pass over the white privilege angle and just talk about the anxiety being felt by the woman or white man walking down the street.
The fears aren't equally reasonable. We're not going to have exactly the statistics we'd want, but as a first pass we can look and see that for every black-on-white homicide there are about five white-on-white homicides, which is pretty close to the black:white ratio in the population (actually it is slightly lower). On the other hand, male-on-X homicide is much more common than any other sort. Rape, at least outside of prisons (which obviously is a different context than walking down the street), is heavily skewed towards male offenders and female victims. Some studies also suggest that female victimization is depressingly common, such that it's not unreasonable for women to be somewhat anxious in many situations where white men wouldn't be anxious and shouldn't be anxious, even if there are scary black people around.
