I don’t think they now your xpub, at least it’s not the case for Ledger Live in which your Xpub is locally encrypted, and you just query certain addresses balances to display your total balance.
I think it’s the same for most wallets actually, but Samourai is different which is why it is controversial sometimes.