Me either, but some optical illusions can be quite harsh. Your brain (not your eyes) is used to doing all of this automatically, and not changing perspectives on a whim, that would be horrible in daily life. I haven't seen black/blue either by the way, and I see all the numbers on those tests correct.I have not once seen the opposite side.
I don't think this is like the spinning silhouette or the shadow on the checkerboard in its entirety... though I believe that it is like that for some people.
http://colorvisiontesting.com/ishihara.htm
http://en.wikipedia.org/wiki/Ishihara_Test
I believe based on the Ishihara Test that there are hard differences in color perception. And for certain people on that spectrum, it must be one, or it must be the other.... but for certain others it depends on how you look at it, like the spinning silhouette, like the "rabbit or duck" illusion, etc.
I don't see how this can have anything to do with your light sensitivity, the hypothesis doesn't make any sense in my opinion. It's easily explained by your brain trying to figure out what color the object is in reality, we do it all the time when the light strikes objects differently.