and about decoding to an array of bytes, both strategies are possible depending on your platform since 24 words is 33 bytes exactly means both approaches produce the same result if using big endian.
Discussion
there is no way around the intermediate stage of 11 bit words spread out... the bech32 encoder does the same thing on bytes, it has an operator that expands the bits and then lookup table to get the characters from the charset