The term "minority" just says, there are fewer subjects with a given characteristic than without it, there is no inherent connection to race. Why attack the generic term if you mean one very specific use case?
Also why is a term describing there are fewer people of a given skin color in an area offensive? Like what's the point?