I set shared_buffers really high. See: https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-SHARED-BUFFERS
They say it should be 25% of your total RAM. But since I have an absurd amount of RAM, I set it to 50% (128GB in my case)
Discussion
I also enabled pg_prewarm so it will populate the cache automatically. Otherwise you have to run queries to populate the cache. https://www.postgresql.org/docs/current/pgprewarm.html