exactly. when you pay someone, 99 times out of 100, you're paying a hash of their public key (xpub), (+ an incremental derivation path, so as to be able to produce many addresses from a single key).
as far as I understand, if you paid their public key directly, you would be able to calculate all of their future receive addresses. just the same way a 'watch only' wallet does, by having you import your xpub.