I can rearrange these so the link doesn't move. But I'd say there are multiple problems to be fixed.

You are following 416 people, that is reasonable, shouldn't be the reason recomputing takes so long.

I doubt this matters but:

Turn on Settings > UI > Show DEBUG statistics in sidebar. How big is "ES"? I'm at 1.1 million.