If you compare it to gold. Somebody might lose their huge stash and someone else could find it later, should we destroy (freeze) the gold?

There's ~21M coins, finding a lost coin is not inflationary in that sense.