I think it's fine to unfollow. If a note is good someone you do follow will boost it back into your feed.

General following quickly becomes hard to track.