That feeling when you suddenly realize you never setup Core Lightning backups when you migrated to a new server...
Thankfully got them setup pretty quickly using the SQLite duplicate database feature to an NFS-mounted NAS over TailScale, but was a terrifying realization nonetheless.
How are you all doing backups for LND/CLN?
😑
With my LND I am just simply running a continuous inotify process to watch for changes to the channel.backup file and having copies made to a remote drive on any changes.
That's a static channel backup and not a full backup as #[4] is describing here.
Right. I am not doing full db backups on my node at the moment. Just have a mirrored drive to protect from single drive failure which is sufficient for my lowly node.
I used to have quite a bit of funds on my routing node but have since toned it down to just a few channels and few million sats for my own spending purposes.
After having to actually go through the nerve-wracking recovery process using the static channel backup, I realized I either need to change my setup drastically or adjust the amount of funds I'm being #reckless with. I chose the latter for now.
How much delay does the backup over tailscale add to your payments you think? Got any estimates?
I would love to do that.
Zero delay as it just means CLN writes to two database directories at once.
Could only see it causing any delay if you had horrendously slow disks, but other than that it's not a bottleneck in any way!
Here are the docs, works well as a systemd service or in Docker:
https://lightning.readthedocs.io/BACKUP.html#sqlite3-wallet-main-backup-and-remote-nfs-mount
Enter #Raspiblitz
CLN:
- the backup plugin replicates the channel database realtime to the OS disk (SDcard on RPi)
- the emergency.recover file (same as LND SBC) is synced to the OS disk and backed up to Nextcloud/Dropbox/Telegram
optionally. The last CLN release can backup with peers as an experimental feature.
LND:
- has no database replication option.
- The SBC is treated as above.
I like using a highly redundant ZFS volume as my data disk.
2 out of 4 disks can fail and be changed on the fly + regular snapshots can be synced to another server or NAS.
♥️
I really need to consider switching to RasbPiBlitz now that you have x86 support...
Thanks to CLN most of these options are easily available and probably easier to configure f not using Docker.
Plus you already added the database replication which can be done parallel to the backup plugin.
The biggest difference is that the backup plugin does the save append-only so better suited for an SDcard (if using an RPi), but also needs compacting (can be done without CLN restart as it is only done on the backup).
nix-bitcoin lets you configure streaming replication to either a second drive or to a remote server https://github.com/fort-nix/nix-bitcoin
Have been meaning to play around with it, looks like a great set of tools.
I’ve been quite happy.
Initial install and config requires a small amount of work but after that incremental changes are a changes to the configuration file and a deploy.
With the peace of mind that if something does not work you can always revert the change and re-deploy .
