Sure you can. White privilege is an assumption about a person because of their race and appearance. It's generally used to undermine, disenfranchise, and dismiss the opinions of speakers that are white. You cannot tell a persons family history or personal experience , or their privilege by their race.