I see what you mean.

Maybe you can raise a feature request and explain what the user experience should be like!

(if you follow a tag it makes sense it shows up in "all follows" so the user experience has to be thought out)