I think I tried it with and without the single ticks and they seemed to be ignored. I believe the index is an integer type in the code, which is why I decided to experiment with the prime marker.

I saw the public_bytes has the prefix on it, making it a 33 byte string instead of 32 bytes. However, changing that didn't fix it. So if that is a problem, it's not the only problem.

I'll make sure my code matches however it is written in BIP32 before I post my code again though, just to reduce confusion.