I'm no expert in any of these areas; but it's often been my experience that upgrading the RAM (if upgradable) will relieve swapping pressure.

Even though your RAM doesn't appear to be even close to maxing out, that doesn't necessarily mean that the memory manager isn't constantly swapping in an attempt to leave as much RAM available for what it can't know might be coming next.

My understanding is that the process of verifying the blockchain is a memory hog and constantly growing worse. A cursory search suggests 8GB is the absolute minimum recommended for syncing a node - and that recommendation could easily already be outdated.

Alternatively - if a RAM upgrade isn't an option - you might want to look at the ubuntu swappiness parameter. Lowering it from the default 60 to more like 20 should increase the RAM usage to swap ratio - for a faster (albeit technically with a potential for being less stable) overall memory space.

Before any of this, however, you might want to use a disk i/o monitor just to verify that your swap drive speed is indeed maxing out.