Whatever was on that address previously is all that’ll be associated together so if it’s unique then nothing is linked.
Unless you’ve leaked your xpub somewhere where they can then associate all these public addresses together but that’s a different story