That's how Ian Malcolm from the Lost World book dressed too! Only grey and black socks so it never matters if you mismatch.

I tried it for a couple years, but people kept saying I looked dreary. I ended up buying long sleeve baseball shirts in different colors as a compromise.

The same, but different.