🙏
Discussion
This is long and rather dense. May be another day before finish. Good start though 👀
Ok, finished reading it. It’s really good and fleshed out! A lot of the framing you gave clarified some vague concepts I held but never considered very carefully. Definitely shifted my schema about knowledge databases a bit.
I gathered you spend a lot of time with graphs and ML. Was curious what you do exactly for say connecting notes with some sematic-weighted distance measure. I know a bit about this, so no need to dumb it down too much if you choose to elaborate. My guess is you leverage some state of the art LLM model and then build a “simple” ML model with some popular distance choice to rank things relative to one another.
Yeah, pretty easy to explain actually. There's a ton of embedding models that you can work with off the shelf if you'd rather not train yourself (can find them on huggingface, a repository of AI models).
Words go in -> numbers come out.
Your vocabulary of words/tokens gets assigned some number values and the training task is to predict some part about the group of words you've been assigned
1) I've masked word or set of words with a blank value, predict the words.
2) predict the next n tokens
etc.
the performance depends on the task and the data it was trained on, but the result is you get a bunch of numbers that you can compare through a bunch of distance metrics. Grab a bunch of text with vectors assigned to them and you ask "which text is closest to this thext i care about", which is some K-nearest neighbors algorithm