nostr:nprofile1qyw8wumn8ghj7un9d3shjtnzd96xxmmfdecxzunt9e3k7mf0qy2hwumn8ghj7un9d3shjtn4w3ux7tn0dejj7qpqutx00neqgqln72j22kej3ux7803c2k986henvvha4thuwfkper4sau8ykj have you tried something that supports tensor parallelism like https://github.com/turboderp-org/exllamav2 ?

The biggest limitation here is the stuff that still has to have CUDA/Nvidia to run. It’s not a huge amount, but enough to create a noticeable friction.

Still really hard not to entertain this TBH 😂😅😬