There is a difference between "as a white person, I felt unsafe around brown people in a particular setting" racism and "as a brown/black person, I feel unsafe in most places and don't feel I can trust authority to look out for my safety" racism. They're both bad, they both have a power imbalance and they both victimize people, but they're different in terms of where, when and for how long they can victimize.
To say that you are "privileged" isn't to say that you're less of a victim of racism during that specific situation or that that kind of behaviour is ok (it isn't), but you're on the whole less victimized by acts of racism throughout your life and within society, and you have more (but not total) control over when and where you get victimized. You have the
privilege to walk away from acts of racism committed against you, and to go to a
safe space where you don't get treated that way. This doesn't mean that the racist acts you experienced were less hurtful, less malicious, or less able to cause something like PTSD.
You felt unsafe at school, but still had the ability to walk away from that racism at the end of the day, just like I walked away from homophobia at my school by not going to it for months at a time. And I feel unsafe around drunk homeless native people because they behave unpredictably and do shout racial slurs at me (in as much as "whitey" is a racial slur, I suppose) but I have the privilege of going home to escape them and laughing about it afterwards. They have no homes and regularly have insults and garbage thrown at them.

Both acts are racist (the latter is probably more classist than racist but racist policies led to them being in the state they're in, by and large), but one form of racism occurs over a longer period of time and is both more violent and less escapable. I am both a victim of racism, and privileged to be able to walk away from racism. The two ideas are not exclusive.
Using the racism of non-white people against other non-white people to justify white people's racism is... also kind of racist.