I still don't understand exactly why it's happening, but the follow history tool in Nostrgram is a good workaround