I despise racism or any sort of bigotry whatsoever, BUT... I think they get a (very) slightly worse rap than they should. Of course, there is simple ignorance and the fact that they have fallen for fear-based propaganda, but they have a nugget of truth.
If the world continues the way it is going, traditional cultures will slowly disappear, along with skin colors. I'm a hard pragmatist with little use for tradition, and I don't care what color the people of the world are, but even I see that it might be nice to preserve some of that.
If we were conserving a savannah and the plains zebras kept mating with the Grant's zebras, I think it would be perfectly natural to want to at least encourage some of the animals to mate with their kind so that we don't lose the line altogether.
Taken with a heaping spoon of salt, I understand their fear.