How do I deal with natural prejudices against me due to my background?

Stop concerning yourself with the judgements of others. Your own thoughts and feelings are paramount. Naturally you will be forced to experience prejudice, but if you allow it to affect your behaviour than you are empowering the people who judge. If you trust your own judgements above others', you will be better off in the end.