Yeah, the danger is being targeted by because you've communicated with someone they deemed objectionable. Whether or not you accidentally leak your nsec that alone could cause some significant pain in your life.