If this is true then an attacker would need my xpub to decrypt?

This is bullish in a way because it will force people not to use addresses twice or leak their xpub. Xpubs will become much more closely guarded data.