Yes, silent payment addresses are like monero addresses: pseudonymous, not anonymous. For best privacy, you should make a new address -- a new pseudonym -- for every payment, otherwise your pseudonym (your address) can be profiled, and its profile can be shared/sold/grown.