Nostr Web Client

I've been tinkering with an LND node for a few months now and the #1 issue is getting out of sync with the chain data. Even when it has the blocks, it sometimes hasn't processed them.

This same problems happened to me with the lightning node in Zeus.

And it's a troublesome problem to detect too because uptime monitors are going to report the service is up. Systemd is going to say the service is running. The blockheight might even be up to date!

It'd take some custom code to dig deep and detect this as soon as it happens. Sure, something could be written, but I haven't come across it yet.

As I understand it, a watchtower is looking for channel closures with old channel state. This is not at all the same as detecting that your node is falling behind and payments are going to fail (sometimes silently) if someone tries to move any money.

The only explanation I can muster for this being the state of lightning after 5+ years is that laws must have had a severe chilling effect and kept developers away. It's hard to understand why the quality would be so low unless there just aren't enough devs to go around. I know the people who are working on these projects are incredibly passionate, so I don't believe for a moment that it's a lack of caring on their part.

Diacone Frost 7mo ago

It used to happen a lot to me as well. I was unable to figure out what's going on.

And in the end it was a failing/slow ssd.

Reply to this note

Please Login to reply.

Discussion

Dr. Hax 7mo ago

Were you getting caught in application startup loops where it would never make any progress in processing any blocks? Trying to figure out if we're running into similar issues.

That one was fixed by upgrading to 0.19.0-beta

Another one was some random remote BTC node going offline and my network connection to it timed out, then my server would gracefully shut down. It looked like an unexpected exception that was being caught, and shutdown is how it was handled.

When I switched the .service file to always restart instead of only on failure, the wallet would be locked. After manually unlocking it, it'd run into an issue, reboot, and start over from the beginning... only to repeat this process.

I hacked around this by turning up the log level to debug and it never happened again. It's been a lot of things like this, where no cause is ever identified and nobody understands why the weird workarounds evade the core problems.

Diacone Frost 7mo ago

I don't remember exactly, but lnd was often out of sync, could not connect to btc node, there were timeouts all over the place.

Actually, better hints gave btc node. When it started up it took forever to load mempool from disk, i2pd showed tons of errors around networking issues.

it was all so random, that I couldn't wrap my head around that.

I figured by accident when trying to copy larger amout of data out of the disk. The speed was good at the beginning but after some time it degraded to kbits/s. smart didn't show any problems either and short tests didn't show any signs.