There is a time and place for violence, but in my opinion any physical force oriented towards punishment is bound to backfire.
Physical force oriented towards ending violence is justified. It's a fine line and very difficult to judge in any general way.