Had a crazy experience early this morning where our main MySQL server shot from 55% disk usage to 99% in about 1 minute. Using ncdu, I couldn't see anything eating up that extra 275GB of disk space. Then it just slowly went back down to 55% usage over the course of about 5 minutes. I have zero explanation for this.

Reply to this note

Please Login to reply.

Discussion

In this situation I would typically run:

lsof | grep deleted

to see if a process had held open a file descriptor for a deleted file. Sounds like whatever it was resolved itself without your intervention, but if it doesn't I know which process I need to stop or restart in order to reclaim disk space.