you don't keep the words in memory all the time, that's just as retarded as keeping the hex encoding in memory all the time when you need to constantly generate signatures with the secret it encodes... yes, you store the damn bits of the secret

the decoding from the word key is human input and then generate the secret and encrypt with a password and voila

the password encryption is literally just hashing the literal password a shitload of times ind some funny complex pattern like in pbkdf or argon/2 and then XORing that with the secret bytes and it's done

you don't decode the damn word key in a way that leaks any timing information because humans are so slow there's no way the nanoseconds of decoding are going to be a part of any side channel of timing