Figured it out. If I click the link in the top right of a note it seems like that's a different representation of a tag feed. If I back up and click the tag link within the note itself I am able to unfollow.