Global Feed Post Login
Replying to Avatar SuiGenerisJohn

Im not an expert but the LLM analyzes the text broken down into little tokens like words or parts of words or letters and then the llms are trained on statistical relationships between the tokens in the data it analyzes. This gives tokens a kind of preference in relation to other tokens (preference is just a word I used not a technical term) so when you use the llm you aren't seeing humanities knowledge youre seeing a statistical representation of humanities mode of expressing its knowledge.

Avatar
Madame President 1y ago

Pretty accurate…the entire calculation is also derived from a relatively simple formula, the cosine similarity, in case anyone was curious

Reply to this note

Please Login to reply.

Discussion

No replies yet.