Curious, what are your thoughts on libbitcoin nowadays? It's designed to address this and other architectural flaws of core.
Discussion
I thought it was interesting but I don't know enough to offer a strong opinion. But these questions would still apply, right? Just quantitative answers would be different, presumably.
(Iirc libbitcoin is faster but more memory hungry.)
It doesnt have the concept of the utxo set or the mempool in memory, just a database of transactions where confirmation status is only one field.
It's also multi CPU core, but I'm not sure how this relates to your question...
Oh really? That sounds a bit crazy, albeit it has its logic .
The multi core part yeah, they focused on parallelisation to speed up IBD. My vague memory from Eric's explanation was that it stuck everything in RAM to make access fast but that's obviously not right.
My question is motivated by the, I hope, obvious intuition that there has to be a limit somewhere where things break, since you theoretically need access to the entire utxo set to complete validation of state updates.
The problem with this is that connecting blocks to tip once you reach steady state will be much slower. Instead of looking up each input prevout from the set of utxos, you will have to lookup each input prevout from the set of all txos ever created.