He has said he felt uncomfortable walking into a theater full of black people to see a movie, and likened it to being in "Planet of the apes" those were HIS words.His words were also mixed people are powerful because they have black bodies and white minds. Those were HIS words.He's not comfortable with black people, and that is not something he has shied away from. YOU seem to feel more shame for his beliefs than he is about them.
I'll tell you how it is though, and what we tell abuse victims. I cannot judge a man based on what he does when its just you two in bed, and I cannot judge a man based on who he is in your head. The ziggurat you place him on in your reality is just that, your reality. It is not the real world, and not in a place we live.
Obviously Joe means something to you, and its not my job to tear down someone you love, but you have to live in reality. Theres the Joe who has NEVER been ashamed of who he is, and there are people like you who obviously are and think what he tells us isnt the real him. He's something more in your eyes, but as the song says, "He's just a man"