They basically get snippets of the prior conversation added automatically to every prompt, at least the last time I checked. The longer a convo goes the more likely it will be to miss important details due to token limits and how many replies its allowed to create to a prompt (splitting attention is demanding?). At least thats how it worked when I was playing with GPT-Neo and GPT-J.

AutoGPT works a bit different.