Let me start off with a disclaimer: I have no idea what it's like to be white. So, as to white privilege, I can't really say. Maybe some white folks feel privileged, and some dont. Perhaps it is a personal thing. If it doesn't exist in your mind, then for you it doesn't exist at all. Others may feel differently.
The core pillar of modern liberalism is victimhood, so it is natural for them to tell people they are victims of something. You can't sell a cure for victimhood if you can't first convince people they are victims. You also can't have a victim without an oppressor, so someone must be labeled as such. Somebody must be the boogeyman, and it's the evil rich, white privilege, Republicans, etc. It's another way to divide and conquer.
They're going to tell you you're either a victim or an oppressor, it's what they do. I wouldn't worry about it. What their opinion is of me has never mattered, and you shouldn't it let their opinion of you matter to you either. Why should it?