Global Feed Post Login
Replying to Avatar liminal 🦠

Yeah, pretty easy to explain actually. There's a ton of embedding models that you can work with off the shelf if you'd rather not train yourself (can find them on huggingface, a repository of AI models).

Words go in -> numbers come out.

Your vocabulary of words/tokens gets assigned some number values and the training task is to predict some part about the group of words you've been assigned

1) I've masked word or set of words with a blank value, predict the words.

2) predict the next n tokens

etc.

the performance depends on the task and the data it was trained on, but the result is you get a bunch of numbers that you can compare through a bunch of distance metrics. Grab a bunch of text with vectors assigned to them and you ask "which text is closest to this thext i care about", which is some K-nearest neighbors algorithm

Avatar
YODL 8mo ago

I see. Again, good stuff. Thanks for sharing

Reply to this note

Please Login to reply.

Discussion

Avatar
liminal 🦠 8mo ago

Thank you 😁 it was a really fun learning and writing it 🎉

Avatar
YODL 8mo ago

Mind if I send you a quick DM? Don't see another way to reach you listed

Avatar
liminal 🦠 8mo ago

Sure!

Thread collapsed
Thread collapsed
Thread collapsed