Hmm... Oh!
Do you have the male bandit badge the chief gave you? If it's in your inventory it might be causing the female bandits to attack you.
Have you tried sleeping at the inn for 7 days? I had to do that to reset Maul's aggro at the male bandits camp, not sure if it's the same though.
Good suggestions, will double-check that later today. Thanks!