the main reason is that computing with physics is really difficult.
create a quantum algorithm is like trying to build a program that consists of throwing stones into a pond and trying to encode the answer in the constructively interfering waves.
that's why there are only like 5 discovered algorithms (limited use cases).
i honestly had no idea how shor came up with it. i mean if you just look at him he's a giga nerd.