Isolate the Redis logs and find out what happened to it. Nostream is unable to connect to Redis in a timely manner.
docker logs -f nostream-cache
I have created an issue for it: https://github.com/Cameri/nostream/issues/346
Any idea how to debug this, nostr:npub1qqqqqqyz0la2jjl752yv8h7wgs3v098mh9nztd4nr6gynaef6uqqt0n47m?
Discussion
Thanks, will try that!
It seems there is an out of memory problem. I have now added 16GB swap to my 16GB RAM, but it still happens. Available Mem (RAM) was mostly 12G, until suddenly:
kernel: [62794.439403] Out of memory: Killed process 1142634 (redis-server) total-vm:27478268kB, anon-rss:11599084kB, file-rss:0kB, shmem-rss:0kB, UID:999 pgtables:47208kB oom_score_adj:0
Does redis really use 27G? Would max mem limit help? How do I set Redis max memory in nostream?
You can reduce the use of memory by removing some rate limits from your configuration as well, that’s all it’s used for.
Thank you! You mean those?
limits.event.rateLimits[].*
These settings can requires GBs of memory on a rather small relay?
Yes, and there’s a few more settings for rate limits.
There’s per events, per message, per connection and per invoice rate limits.
The default rate limits might consume too much for your relay. Try removing some connection rate limits first and see how it goes.
Great, I will try that.
Would it be okay for the app to use the following redis config:
maxmemory 2mb
maxmemory-policy allkeys-lru
From the docs: "all the keys will be evicted using an approximated LRU algorithm as long as we hit the 2 megabyte memory limit."
Source: https://redis.io/docs/management/config/#configuring-redis-as-a-cache
Yes, your rate limits with long periods may become ineffective though with such a strict memory limit of 2mb. It may make sense to just get rid of them and just use short-period rate limits.
Now I understand it, thanks. I just realized that I have some rather old default limit in my settings, that is not in the current default-settings.yaml with a duration of 1d. First I'll get rid of this, let's see if it helps. I will document your tips in my github issue. Maybe someone else is having similar problems.
It seems I am back to square one: After removing rate limits no out-of-memory and no redis restarts had happened again. But still my invoice page goes offline now and then. No apparent entries in any logfile (nostream, nostream-cache, nostreamdb). 🤔
Enable debug logging on Nostream by setting the DEBUG env var to `worker:*`
Then go to your invoice page and inspect the logs.
Wow, this is a dense log! Thanks, I'll report back!
Oh no, this is embarrassing: I think I've finally found the problem and it has nothing to do with nostream. 🙈 It seems the main reason of my outages have been too low limits in my NGINX configuration (worker_connections & worker_rlimit_nofile). But thanks to your help I have improved swap and memory efficiency on the way. 🤙 I am 99% sure the problem is gone now. I'll wait some days and then document my findings in the issue and close it. Thanks for your help!
Glad you were able to solve it! 😎 If you get any more issues let me know.
That would be useful to mention in the README
You could remove per event rate limits altogether and just keep per message rate limits, for example.