Honestly, the first AC is the one I have the best memories of.
I've played up to Black Flag and I agree with this. AC2/Brotherhood are the most polished and fun but they're kind of typical decent finish and forget Ubisoft games. AC1 actually tried something new and interesting even if it mostly failed at it