Thanks for the recommendation, but from the reviews and previews I saw, it looks like it might lack enough detail for what I want.

Ideally I want to be learning how to do the math myself.

I'm considering going through Secret Key Cryptography by Frank Rubin