I chose correctly the first time, but it was all luck.
when you take the quest, you have the option of heading to map markers either in the north, or backtrack to one further south. head to the northern one, and the southern marker will disappear, and point you to the (incorrect) female bandit camp.
the same thing happens if you head south..you'll get a tip pointing you south and the northern markers vanish. Is it a bug? or the devs just giving you one shot at the location?
I had both choices too but that pawn I had told me it was at the southern camp I could find the book. When your pawn has a red "?" above their head you can ask them about the quest your doing.