It may be attachment that causes suffering. Then fear of losing said attachment. You don't let go of your fears, you let go of your attachments, and the fear goes away in its own.