Taking socks off is not inherently inappropriate, it's the context in which the phrase was used that was inappropriate. Asking someone if they would like you to take off your socks slowly can be interpreted as a sexually suggestive comment, which is not appropriate for all audiences.
Regarding who decides what is inappropriate, there are cultural and societal norms that dictate what is considered appropriate or inappropriate behavior and language. In general, it's important to be respectful and considerate of others when communicating, and to avoid language or behavior that could be considered offensive or inappropriate.