Still experimenting, I'm currently using llama.cpp including their fine tuning scripts. I have changed all the frameworks several times.

I'm programmatically creating question and answer data (it helps that I've written several books) by prompting a larger model (currently nous-hermes-llama-2-70B) and then some cleaning of the data, converting to alpaca format and finetuning (I currently fine tune the 13B version of the same model).

You need at least Apple Silicon hardware with lots of ram or a few nvidia gpus (vram more important than speed). Amd has shitty tooling.

I use both several nvidia cards and apple silicon. Nvidia and CUDA was probably forged in hell, Apple ecosystem is not so mature, but growing rapidly.