Destroying the drive is always the answer if you have sensitive information on it. I suppose you just need to decide how sensitive you think your xpub is, since it doxxes the size of your stack, which could make you a target.

Anyone who recovered your xpub from that drive would know exactly how much Bitcoin you have associated with the private key on your Jade, and they would know that you have a hardware wallet, since the keys were not on the laptop. That would give them a significant amount of information to go on if they decided it was worth breaking in to find your hardware wallet.

That's without taking into consideration any other sensitive data you may have had on your laptop that hasn't been a part of this conversation already.