As a negative injunction, the harm principle is extremely weak and vague as a system of self governance, and you're right: a materialist principle ignores whole realms of moral hazard. And there's no vision! In the end, the strong do what they will, and the weak suffer what they must.