Thanks for the recommendation, but from the reviews and previews I saw, it looks like it might lack enough detail for what I want.
Ideally I want to be learning how to do the math myself.
I'm considering going through Secret Key Cryptography by Frank Rubin