Handbrake is awesome. If you are planning on using either Plex or JellyFin, you'll need h264. Most devices you will play back on support h264, so it won't have to be live transcoded for playback.
I ran a rather large Plex setup, let me know if you have questions! ♥️🐈⬛