There is no hidden assumption. Any use of violence (except in defense against an aggressor) is a violation of human rights.
When confronted with aggressors, it is necessary to defend yourself. This can be done by defending yourself directly or voluntarily agreeing to a defensive service.
I believe you misunderstood my position, as I’m not claiming that violence is “worse” than any other human rights violation. Rather, the key point is that violence is always a violation of negative rights, which makes it inherently illegitimate, except in self-defense. This is where I see the contradiction in your stance: if you use violence to achieve another human right, you are violating another person’s right in the process. This creates an inherent contradiction in the approach.
Lastly, I want to clarify that I am not casually throwing around terms like ‘libertarian.’ I have carefully defended my position and believe I understand the underlying principles. I’m not attempting to impose my views on you, just sharing what I believe is a sound moral stance.