it depends on the context/tone of the conversation. No one likes to be referred to as white boy/black boy, sometimes it just doesn't feel like its all in fun and games; it really can be hurtful and offensive - especially to a grown man/woman.
Honestly I've made reference to 'that white guy' when pointing out someone, and I've been referred to as 'who? that black girl?'. that is harmless IMO, because i don't take it as anything negative. But if you've experienced negative racial situations, even in simple situations these words seems to be magnified. You have to acknowledge that alot black people find it offensive more so because of the history attached to those words(This is just fact, some black people are still resentful).
Honestly if it comes off as being derogatory from ANYONE (black or white) then you have every right to feel offended.