No idiot, the legality depends on whether the owner still actually owns it and can demonstrate that he does. The feasibility of him maintaining control of it then depends on his defense, which can be purchased on the market. He may cease to care and he may relinquish the property. But he won't always.
Discussion
If the NAP is followed and he still holds on to the property and you find this to be against your legal theory, then you are not really following the NAP. If that is true then you want anarchy, but only this special way, and you're willing to permit aggression to get it, unless your theory stipulates this as a preferred norm for voluntary association, and not an absolute.