I'm not making any moralistic judgments here, but the issue comes down to property rights.
If you enjoy the benefits of property rights (everyone does whether they realize it or not) then it's best to respect the property rights of others.
Of course there are always gray areas. That's up to you to determine.