It's been a long time since I spoke Prolog in anger, and I don't recall the syntax properly, but here's something that seems to work:
mySetMember(X,[H|R]) :- equalSetOrAtom(X,H).
mySetMember(X,[A|Y]) :- mySetMember(X,Y).
equalSetOrAtom(X, X).
equalSetOrAtom(A, X) :- containsSet(A, X), containsSet(X, A).
containsSet([], X).
containsSet([H|T], X) :- mySetMember(H, X), containsSet(T, X).
The reasoning here is that your check for equality in the first definition of mySetMember() doesn't account for unordered sets. Hence the not-particularly-efficient check for equality between two unordered lists.
Note that this will only work if you don't mind [a,b] being seen as equal to [b, a, b, b, a, b, a, a, b]. If you need that, then you'll have to be a bit more cunning.
Note that this almost certainly isn't the most efficient way of doing things, because it's late and I'm tired.