I bought an MI100 recently for $650. 32GB of HBM2 and it performs around around 0-5% faster than a 3090 on the default flash attention 2 benchmarks. Performance on actual applications can be mixed though, as many are not well optimised for CDNA's matrix cores - even where work has been done for RDNA, which is not that often, it doesn't necessarily carry over. It's also frustrating when efforts to improve performance get turned back by maintainers: llama.cpp closing PR for flash attention on AMD because the requisite (header-only) lib is supposedly adding an unneeded dependency (https://github.com/ggerganov/llama.cpp/pull/7011).
There's also a few tricks/updates I'd like to try which may improve performance, e.g. hipblaslt support being added next rocm release - of course these are "maybes".
To give you a rough idea of practical performance, default SDXL with xformers is around 4.5-5it/s (between 3090 and 4090 from my understanding), and exllamav2 with qwen 72B at 3bpw is around 7t/s (slower than a 3090, though a 3090 has to use a lower precision to fit).
As others have pointed out, I can't really see what this project offers for AMD users over existing options like llama.cpp, exllamav2, mlc-ai, etc. Most projects work relatively easily these days.
There's also a few tricks/updates I'd like to try which may improve performance, e.g. hipblaslt support being added next rocm release - of course these are "maybes".
To give you a rough idea of practical performance, default SDXL with xformers is around 4.5-5it/s (between 3090 and 4090 from my understanding), and exllamav2 with qwen 72B at 3bpw is around 7t/s (slower than a 3090, though a 3090 has to use a lower precision to fit).
As others have pointed out, I can't really see what this project offers for AMD users over existing options like llama.cpp, exllamav2, mlc-ai, etc. Most projects work relatively easily these days.