One of the coolest and creepiest questions for LLM research is how many prompts the LLM takes to figure out exactly who the user is.

Most LLMs can already identify the user's gender, age range, education, and socio-economic status from prompts alone and reply accordingly. Current accuracy is about 80% after 6 prompts.

More interestingly, how should the LLM reply if it wants to trigger the user into revealing who they are so that the LLM can answer questions more precisely.