Yes, you do not gain any privacy from the sender but only from third parties trying to identify payments between you two. This is very valuable for donations to NGOs for example. If they posted a static permanent address on their website, a evil government could look up that address and see how much was donated to that party/NGO and where the funds came from i.e. who donated.

If you want to read some more about it:

https://bitcoinmagazine.com/technical/bitcoin-silent-payments-secret-keys