Good point. I can only guess that its a state matter only and the constitution doesen't give power to make laws restricting free association within the boundaries of a state. This could be a much bigger deal than just affirmative action, if taken to an extreme, it could unravel 90% of the federal government.