We all also operate on different levels of "whiteness". A lot of people who are considered properly "white" in one context will in other contexts be targeted & excluded due to some characteristic that makes them not the "right" kind of "white".
Whiteness is a construct for global oppression, crafted over centuries. It's complicated. However, if you are prickling at the idea of "dealing with your relationship to colonialism", I would suggest maybe there is something there you need to look at.