there is a second monotonic index i'm adding to the key value store that relates to a new key type, which i'm calling a "pubkey index"
this is only a key, and contains the full pubkey and the monotonic index connected to it
in place of the pubkeys in the database form of the event the index is substituted for the key and then when it's read, the index is fetched and replaced before comparing it or encoding it based on a search picking it up in a request
i have bumped into a minor problem of order of operations just now, the index keys cannot be generated without the pubkeys (they are part of the index keys, so i have to rethink where this compression is going to occur because i had tried to avoid putting it through the full interface and containing it in one part of the procedure but i have just seen from a stack trace that it fails to generate the index keys due to the substition being done before it
so i have to figure out where to put the index key generation somehow into the interface, it's quite a breaking change in the implementation... i put the indexing operation before the save operation and it has to be done during the save
ah yeah, i hadn't fully thought through this implementation but what i essentially have to do now is add a step in the index keys generation where it fetches the pubkeys it finds
well that's one way, sorta arse-backwards because now it's going to swap them out, then need to search for what was just saved out to get the keys
so, yeah, more thought required yet before i have nailed this one down, the interface is a hard boundary that constrains how i can do this
oh, this is awesome
it has also now exposed that a heap of scanning and index generation work is happening inside of the database threads which means that i may gain an extra boost of processing performance out of this because it will generate the indexes before and outside of the database view/update threads meaning the database will have more time to do its work of writing, reading and compacting logs
there may be a net gain in throughput out of this, but i am now certain that it will not be a loss, and it's also triggered me to start looking... gonna go investigate all of the database operations now and move the heavy computation or memory shuffling parts outside of these threads because this will dramatically boost the performance of the database and shift that work into the query threads
i had a few small issues when i was writing GUI code that instead of definding data types uses functional architecture to execute a collection of operations constructed ad-hoc to define the render queue - in this way of designing GUI, known as Immediate Mode GUI (imgui is the first of this type, but really this is rooted back in the design of Newsqueak, which is the real great grand-daddy of Go) you have to avoid doing processing in the render queue to avoid it triggering blocking in this thread, and for which reason newsqueak (and Go GUIs) need to shift the model out of the view part, to use MVC model... the model is the data, the controller is execution, and the view is the render, the render thread needs to be clean and do minimal computation, only as much as is required to turn the data into a paint
i got it to work
literally just took moving a couple of calls to the event index generation to before the marshal function... wss://test.realy.lol is now functioning with a full clone of the data taken from wss://mleku.realy.lol
i can commit this now
Thread collapsed
Thread collapsed
Thread collapsed
Thread collapsed
