Your new contact list isn't spread to all relays and it could happen that you received an old contact list and followed someone and your client updated it as the new one, bringing back the npub you had unfollwed.

One way to keep track of your contact list could be to see metadata.nostr.com