Well two points to this:
First, that's kinda a weird way to answer the question. If you ask what color something is and then show a picture, reality shouldn't matter. What color is this rectangle?
Nobody's going to say "red" even though that's what it was before I inverted the colors. I agree that some people are answering the question that way. I just think answering the question that way is objectively wrong. You could add a qualifier like "outside of the lighting conditions, the dress is..." but that doesn't change how the dress appears in the image.
Second, I think most people who are answering blue/black actually see those colors.
To those people, I whipped this up real quick:
I took the default MS Paint Black color and a brownish/gold color that I kinda randomly picked using "edit colors" tool. The black band across the dress really looks like the top and nothing like the bottom to some people? I think everyone can agree the object on the left side of the dress is black.