I feel offended when this happens...?

First off, I am 18, short and slim, and my ethnicity consists of 1/2 Japanese, 5/12 Korean, and 1/12 Black which brings me to this issue. People are always asking me my ethnicity and I always tell them what I just said or I'll just say Korean, Japanese, and Black. When I say "Black", I get these weird looks like "what?" and "You've got to be kidding me!" It's because I don't look Black AT ALL. My skin is light, I have Asian eyes and the Asian hair texture and whatnot. After I get those looks the next thing usually said or thought is "you just wanna fit in and be something you're not" which really hurts me for the fact that they don't even know me or my family enough to actually know that I am mixed with Black. I have been called racist for this for some odd reason. I don't see it as racist. So, to avoid feeling offended by others comments when they ask my ethnicity, should I just mention only Japanese and Korean?