If you stay away for a while or visit at unusual times, they’ll probably move on.