Our Mastodon server stopped working after around 22:37 UTC time yesterday. It appears Mastodon spun out of control consuming lots of CPU, disk and memory. It locked up the server with a massive amount of I/O. There isn't a lot of information about what happened since logging stopped persisting to storage shortly after it started. It was quickly detected by our service monitoring but we just weren't around to fix it for 7 hours. It's working again after a server restart but nothing was changed.
Discussion
OS release announcement we posted several hours earlier never got processed and actually posted, so it appears something was wrong with it for at least around 4 hours before it stopped responding to requests. It seems to be working fine now.
We're not sure what happened. It might have been a small scale denial of service attack of some kind which started disrupting the service and took a while to fully take it down...
We posted this OS release announcement again:
Mastodon look too centralized to me. Nostr don't have problem with just one relay down. Posting on multiple relay si standart and so why we can have private relay when its down we still get notes
I would make it a habit to reboot once a week.
#Nostr "account" unaffected
Mastodon server goes down: 😱
Nostr relay goes down: 😏
Kinda…
nevent1qqs0h9glh5rnjskt39n4fgnnjt9xkf0c6r9h2k6g6qtc3feef7pca7czypsm77gtyz22lvp5jhy7zd4v7c2mur7vct9etddvld0kenh7rzcxyqcyqqqqqqgpzemhxue69uhhyetvv9ujumn0wd68ytnzv9hxgqgswaehxw309ahx7um5wgh8w6twv5q3gamnwvaz7tmjv4kxz7fwv3sk6atn9e5k7qgmwaehxw309akx7em9dchxyarrvehhyurvv438xtnrdakszrthwden5te0dehhxtnvdakqz8rhwden5te0dehhxarj9e3xjarrda5kuetj9eek7cmfv9kqzyrhwden5te0dehhxarj9ekxzmnyqyt8wumn8ghj7un9d3shjtnswf5k6ctv9ehx2aq396nky
