The only way to fix this is to flood those hashtags with actual stuff you care about and mention people who you know are also into this to tag their posts as well.

I love Star Trek but I'd probably not tag it in my posts if I talked about here nor would I follow the topic.

The Mastodon/Fediverse modus of operandi of having instances be theme-focused makes community building around niche/fan/nerd topics easier but comes at the price of less freedom of speech because everyone wants safe spaces now.