Replying to Bogi

Bitcoin core by default only keeps around 450mb off the utxo set in ram. So I'm confused about what you're trying to accomplish.
Avatar
Dr. Bitcoin, MD 3w ago đź’¬ 1

I see what you are getting at with dbcache…but I don’t think that limits size of utxo set in ram (or system cache thereof)…

Anyhow, constantly swapping from ram to disk in random fashion is really slow; as many entries in the utxo set will never be used again, these can be offloaded from ram and decrease swapping.

We know the order utxo will be used during IBD, so it is possible to store these sequentially on disk and convert random swaps in to sequential ones.

Try syncing a 1 or 2 GB rpi from Genesis. Or even just reverifying the chain with that little ram and the problem becomes evident…I’m not sure such little ram even works for ibd anymore.

Reply to this note

Please Login to reply.

Discussion

5f
Bogi 3w ago đź’¬ 1

Just watch the memory usage. It never really gets much over 1gb with the default setting. So there's no way it's keeping the whole utxo set in ram.

At a certain point the bottleneck is not ram swapping (I think going above 16gb dbcache has deminishing returns) but the single thread CPU performance. I don't think you're going to get a performance boost from pruning the utxo set if that's when possible. So even if your pi has 16gb of the ram the poor single thread CPU performance will be the main bottleneck.

Avatar
Dr. Bitcoin, MD 3w ago đź’¬ 1

So utxo bloat isn’t a big deal? My understanding was that the Utxo set that gets freed from ram is cached by OS and loads essentially instantly when enough unused ram is available for cache, but if you run out of ram and have to free cache that loading from disk takes much longer…I could be wrong.

When I run top on my node, available ram drops over time until there’s less than 1 GB of 15 GB free…but the OS could be caching other stuff too.

5f
Bogi 2w ago đź’¬ 1

I don’t think the utxo bloat is as big a deal as it’s made out to be. In my way of thinking… you’re admitting that Bitcoin has a bad design if you think it is a big deal.

During initial block download the loading and unloading of the utxo set from the hard drive is becoming trivial. Segwit and Taproot have now made verification of the chain more cpu intensive. If you think utxo bloat is a very big deal then I think you need to consider those two changes as being bad decisions for Bitcoin. They will save you storage and keep the full size of the blockchain data smaller in exchange for increased cpu usage. I think that is a bigger deal than storage space.

If you’re running a public node or a private node for a business and you need to be able to process/query LOTS of transactions then maybe storing the full utxo set in ram will be beneficial. If you’re running a personal node then I don’t think running the full utxo set in ram is necessary.

What you’re basically looking for is a shortcut, skipping proof of work for the node… i.e. checkpoints. There are several already included in Bitcoin Core.

Have you looked into this project? It’s doing a shortcut on the utxo set to bootstrap IBD.

https://bitcoinops.org/en/topics/utreexo/

Also the project led by Eric Voskuil, libbitcoin, is a full Bitcoin node server/client/wallet that is designed from the ground up. I think it might have the ability to keep the full utxo set in ram. Still a work in progress… they’ve been working on it over 5 years maybe closer to 10.

https://github.com/libbitcoin

Avatar
Dr. Bitcoin, MD 2w ago

Good points. Keeping utxo set in ram probably matters most to miners. And requiring 64 or 128 GB of ram in 10-20 years probably isn’t a huge ask (compared to data center costs).

I’ve spoken with Tadge IRL about utreexo at MIT earlier this year. As best I could understand, still waiting for a trustless way to initialize the accumulator. But way over my head