Good question, and I think at the root of it, you may incidentally be right.

There is an american? saying / social convention - "don't wear white after labor day". I associate this with being originated by rich socialites in New England communities like the Hamptons.

Might have more to do with fall rains creating muddy conditions that would soil white clothes

I doubt anyone taking this convention seriously would be doing much laboring.

I would see the practice of wearing white clothes as symbolic of not laboring, as well as being privileged to exercise the option not to wear white, implying enough wealth to have a seasonal wardrobe - not generally something the laboring classes would have.