i don't think you can escape the sexual connotations of the word, considering the word's origin and the way it's applied these days, as haphazard and overused as it may be.Apologies, it's a personal thing - I object to the overuse of the word "loli" to describe all little girls in anime because to me it implies a sexual element to the description (I know, I know, goth loli has no such implications etc. etc. but that's how it is to me).
So you're right - I think within anime fandom it's not an inherently sexual term. But to me, it is.
sometimes a little girl is just a little girl.