Not true. For example, i once took a 401k loan out to buy cattle which fell through so rather than paying it right back into my 401k to close the loan, i bought bitcoin.
I didnt have access to this money, but via the loan i was able to use it for something more productive without incuring penalties and the interest is paid to myself.