Is my goldfish a white supremacist?

I bought a whitish goldfish. And I also have black and white pebbles in the tank he swims in. And for some reason, he tends to stay away from black pebbles. Should I be concerned about my goldfish's bigotry?