Performance of llama.cpp with Vulkan #10879
Replies: 83 comments 127 replies
-
|
AMD FirePro W8100
|
Beta Was this translation helpful? Give feedback.
-
|
With the latest updates:
|
Beta Was this translation helpful? Give feedback.
-
|
AMD RX 470
|
Beta Was this translation helpful? Give feedback.
-
|
With the latest updates:
|
Beta Was this translation helpful? Give feedback.
-
|
ubuntu 24.04, vulkan and cuda installed from official APT packages.
build: 4da69d1 (4351) vs CUDA on the same build/setup
build: 4da69d1 (4351) |
Beta Was this translation helpful? Give feedback.
-
|
Macbook Air M2 on Asahi Linux ggml_vulkan: Found 1 Vulkan devices:
|
Beta Was this translation helpful? Give feedback.
-
|
For the record I think this is slow on the HoneyKrisp side rather than llama.cpp |
Beta Was this translation helpful? Give feedback.
-
|
Can you share how you got vulkan to build on Asahi? I can't seem to get cmake to notice it. |
Beta Was this translation helpful? Give feedback.
-
|
Spoke too soon, got it working! |
Beta Was this translation helpful? Give feedback.
-
|
Gentoo Linux on ROG Ally (2023) Ryzen Z1 Extreme ggml_vulkan: Found 1 Vulkan devices:
|
Beta Was this translation helpful? Give feedback.
-
|
ggml_vulkan: Found 4 Vulkan devices:
|
Beta Was this translation helpful? Give feedback.
-
|
Cool setup! Could you also post the result of 1, 2 and 3 7900 XTX GPUs? You can use only the first GPU with |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
For this multi GPU case getting Vulkan to support #6017 pipeline parallelism might help improve the prompt processing speed. |
Beta Was this translation helpful? Give feedback.
-
|
@netrunnereve I updated the commit id in all my result. |
Beta Was this translation helpful? Give feedback.
-
|
build: 0d52a69 (4439) NVIDIA GeForce RTX 3090 (NVIDIA)
AMD Radeon RX 6800 XT (RADV NAVI21) (radv)
AMD Radeon (TM) Pro VII (RADV VEGA20) (radv)
Intel(R) Arc(tm) A770 Graphics (DG2) (Intel open-source Mesa driver)
|
Beta Was this translation helpful? Give feedback.
-
|
@netrunnereve Some of the tg results here are a little low, I think they might be debug builds. The cmake step (at least on Linux) might require |
Beta Was this translation helpful? Give feedback.
-
|
I've added My release numbers for the RX 470 are basically identical to the ones I posted earlier without the flag.
|
Beta Was this translation helpful? Give feedback.
-
|
Maybe not in your case, but some other results are suspiciously low in tg (for example the RTX 3080) |
Beta Was this translation helpful? Give feedback.
-
|
Build: 8d59d91 (4450)
Lack of proper Xe coopmat support in the ANV driver is a setback honestly.
edit: retested both with the default batch size. |
Beta Was this translation helpful? Give feedback.
-
I put my A770 into a Windows PC and gave Intel GPA and vtune a shot: GPA just crashes most of the time, I couldn't get it to trace anything useful. vtune works, but does not support Vulkan. It just shows some high-level metrics in that case, not really useful sadly. |
Beta Was this translation helpful? Give feedback.
-
I did build it with cmake with build type Release. |
Beta Was this translation helpful? Give feedback.
-
|
In that case it's something else, cause it should be performing similarly to my A770. I suspect the mesa version, there was something in newer mesa versions that slowed down tg on Intel. |
Beta Was this translation helpful? Give feedback.
-
|
A750 has 448 CUs, A770 has 512 CUs I think. Personally, I am not worried about tg. I am worried about pp here. The gemm batch quickly saturates my GPU. |
Beta Was this translation helpful? Give feedback.
-
|
@0cc4m https://gitlab.freedesktop.org/mesa/mesa/-/issues/12585 |
Beta Was this translation helpful? Give feedback.
-
|
Here's something exotic: An AMD FirePro S10000 dual GPU from 2012 with 2x 3GB GDDR5. build: 914a82d (4452)
|
Beta Was this translation helpful? Give feedback.
-
|
Very interesting, and looks like it's pretty close to the W8100 in tg despite being a dual GPU card. Your backend scales pretty well with layer splitting which is why I find it worthwhile to run my RX470 and W8100 together (I end up getting results that are close to the average of both cards).
|
Beta Was this translation helpful? Give feedback.
-
|
Latest arch with For the sake of consistency I run every bit in a script and also build every target from scratch (for some reason kill -STOP -1
timeout 240s $COMMAND
kill -CONT -1
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Iris(R) Xe Graphics (TGL GT2) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | warp size: 32 | matrix cores: none
build: ff3fcab (4459)
This bit seems to underutilise both GPU and CPU in real conditions based on
|
Beta Was this translation helpful? Give feedback.
-
|
Unless you reduce the number of GPU layers, threads and openblas/non-openblas is not gonna make any difference. Try it with ngl 0, then only prompt processing is accelerated using Vulkan, the rest runs on CPU. This is often a good setting for integrated GPUs. |
Beta Was this translation helpful? Give feedback.
-
|
That's something I didn't think about, with
build: ba8a1f9 (4460)
|
Beta Was this translation helpful? Give feedback.
-
|
It seems latest patches has improved the results a bit: ggml_vulkan: Found 1 Vulkan devices:
|
Beta Was this translation helpful? Give feedback.
-
|
Intel ARC A770 on Windows:
build: ba8a1f9 (4460) |
Beta Was this translation helpful? Give feedback.
-
Single GPU VulkanRadeon Instinct MI25 ggml_vulkan: 0 = AMD Radeon Instinct MI25 (RADV VEGA10) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
build: 2739a71 (4461) Radeon PRO VII ggml_vulkan: 0 = AMD Radeon Pro VII (RADV VEGA20) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
build: 2739a71 (4461) Multi GPU Vulkanggml_vulkan: 0 = AMD Radeon Pro VII (RADV VEGA20) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
build: 2739a71 (4461) ggml_vulkan: 0 = AMD Radeon Pro VII (RADV VEGA20) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
build: 2739a71 (4461) Single GPU RocmDevice 0: AMD Radeon Instinct MI25, compute capability 9.0, VMM: no
build: 2739a71 (4461) Device 0: AMD Radeon Pro VII, compute capability 9.0, VMM: no
build: 2739a71 (4461) Multi GPU RocmDevice 0: AMD Radeon Pro VII, compute capability 9.0, VMM: no
build: 2739a71 (4461) Layer split
build: 2739a71 (4461) Row split
build: 2739a71 (4461) Single GPU speed is decent, but multi GPU trails Rocm by a wide margin, especially with large models due to the lack of row split. |
Beta Was this translation helpful? Give feedback.
-
|
What is the power profile for this MI25? Mine is 110W but its running slower than yours on git from today. |
Beta Was this translation helpful? Give feedback.
-
|
Mine defaults to 220w. |
Beta Was this translation helpful? Give feedback.
-
|
AMD Radeon RX 5700 XT on Arch using mesa-git and setting a higher GPU power limit compared to the stock card.
I also think it could be interesting adding the flash attention results to the scoreboard (even if the support for it still isn't as mature as CUDA's).
|
Beta Was this translation helpful? Give feedback.
-
|
There is no Vulkan flash attention support (except with coopmat2 on very new nvidia drivers). What you're measuring here is a CPU fallback. |
Beta Was this translation helpful? Give feedback.
-
|
I see, I was sure about the CPU fallback but didn't know there was no flash attention support at all. |
Beta Was this translation helpful? Give feedback.
-
|
I tried but there's nothing after 1 hrs , ok, might be 40 mins... Anyway I run the llama_cli for a sample eval...
Meanwhile OpenBLAS |
Beta Was this translation helpful? Give feedback.
-
|
Even at below 1t/s |
Beta Was this translation helpful? Give feedback.
-
|
Truth is ... (0.79 tokens per second), 3788.63 ms / 3 tokens So it's not even...it just slower... |
Beta Was this translation helpful? Give feedback.
-
|
OS: openSUSE Tumbleweed
build: de4c07f (5359) |
Beta Was this translation helpful? Give feedback.
-
|
That would be cool to have a graph showinv cuda vs vulkan performance over time/versions |
Beta Was this translation helpful? Give feedback.
-
|
I've noticed that on my RX 7800 XT, the performance of the RADV driver is significantly worse than AMDVLK when using coopmat. In fact, the integer dot implementation ends up being much faster. Has anyone else run into this? It seems like it could be a driver implementation issue, but I’d like to gather some feedback before diving deeper. COOPMAT RADV
ggml_vulkan: 0 = AMD Radeon RX 7800 XT (RADV NAVI32) (radv) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
COOPMAT AMDVLK
ggml_vulkan: 0 = AMD Radeon RX 7800 XT (AMD open-source driver) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
INT DOT RADV
ggml_vulkan: 0 = AMD Radeon RX 7800 XT (RADV NAVI32) (radv) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: none
INT DOT AMDVLK
ggml_vulkan: 0 = AMD Radeon RX 7800 XT (AMD open-source driver) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: none
build: 360a9c98 (5379) |
Beta Was this translation helpful? Give feedback.
-
|
I'm not sure why it does report a different amount of shared memory. I tried running the bench from a clean linux distro and the results are the same. However I don't see how that would only affect the coopmat usage. |
Beta Was this translation helpful? Give feedback.
-
|
I don't think that makes a difference here, at most the largest size for MoE wouldn't fit, but on AMD the large tiles are disabled for performance reasons anyways. The medium tiles stay at less than 10KB. |
Beta Was this translation helpful? Give feedback.
-
|
If you have more shared memory you can fit more waves on the core, but that shouldn't cause such a big difference. Honestly I wonder how well radv has optimized coopmat considering how it's not used for graphics purposes, and afaik we're one of the few programs that support it. I guess I can say the same thing for Intel graphics as their coopmat implementation performs terribly. |
Beta Was this translation helpful? Give feedback.
-
|
I've just opened an issue on mesa's gitlab repo. |
Beta Was this translation helpful? Give feedback.
-
|
Same behavior on my (underclocked) 7600 XT: radv, mesa 25.0.4-1~bpo12+1ggml_vulkan: 0 = AMD Radeon RX 7600 XT (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
build: 0d5c742 (5443) amdvlk 2025.Q2.1ggml_vulkan: 0 = AMD Radeon RX 7600 XT (AMD open-source driver) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
build: 0d5c742 (5443) |
Beta Was this translation helpful? Give feedback.
-
parameter --device performance effect on multi GPUs configvulkan-1.4.314 + llama.cpp Build 5395 9c404ed on Mac Pro 2019 + 8 GPUs Qwen3-235B-A22B-Q4_K_M (142 Go) Run 1 with --device parameter llama_perf_context_print: prompt eval time = 4451,51 ms / 31 tokens ( 143,60 ms per token, 6,96 tokens per second) Run 2 without --device parameter llama_perf_context_print: prompt eval time = 11306,90 ms / 29 tokens ( 389,89 ms per token, 2,56 tokens per second) |
Beta Was this translation helpful? Give feedback.
-
parameter --flash-attn performance effect on multi GPUs configvulkan-1.4.314 + llama.cpp Build 5395 9c404ed on Mac Pro 2019 + 8 GPUs Qwen3-235B-A22B-UD-Q5_K_XL (167 Go)./llama-cli -m Qwen3-235B-A22B-UD-Q5_K_XL-00001-of-00004.gguf -mg 6 -ngl 99 --no-mmap -p "Using one single html script, create a beautiful website" -c 13072 llama_perf_sampler_print: sampling time = 6,35 ms / 74 runs ( 0,09 ms per token, 11659,05 tokens per second) ./llama-cli -m Qwen3-235B-A22B-UD-Q5_K_XL-00001-of-00004.gguf -mg 6 -ngl 99 --no-mmap --device Vulkan4,Vulkan5,Vulkan6,Vulkan7,Vulkan3,Vulkan2,Vulkan1,Vulkan0 -p "Using one single html script, create a beautiful website for a tutorial on Tensorflow" --rope-scaling yarn --rope-scale 4 --yarn-orig-ctx 32768 -c 13072 -fa -ctv q8_0 -ctk q8_0 llama_perf_sampler_print: sampling time = 140,92 ms / 1007 runs ( 0,14 ms per token, 7145,70 tokens per second) ./llama-cli -m Qwen3-235B-A22B-UD-Q5_K_XL-00001-of-00004.gguf -mg 6 -ngl 99 --no-mmap --device Vulkan4,Vulkan5,Vulkan6,Vulkan7,Vulkan3,Vulkan2,Vulkan1,Vulkan0 -p "Using one single html script, create a beautiful website for a tutorial on Tensorflow on MacOs with metal gpu" -c 13072 --flash-attn -ctv q8_0 -ctk q8_0 llama_perf_sampler_print: sampling time = 402,09 ms / 2812 runs ( 0,14 ms per token, 6993,49 tokens per second) |
Beta Was this translation helpful? Give feedback.
-
|
7900 XTX Vulkan ./build/bin/llama-bench -m '/mnt/gamu/AI/textModels/llama-2-7b.Q4_0.gguf' -ngl 100
build: c9c64de (5431) Vulkan -fa 1 ./build/bin/llama-bench -m '/mnt/gamu/AI/textModels/llama-2-7b.Q4_0.gguf' -ngl 100 -fa 1
build: c9c64de (5431) ROCm ./build/bin/llama-bench -m '/mnt/gamu/AI/textModels/llama-2-7b.Q4_0.gguf' -ngl 100
build: c9c64de (5431) |
Beta Was this translation helpful? Give feedback.
-
|
Can you please test Gemma 3 27B for me? I'm interested is it worth to get Mac Studio for it or use 7900 XTX. |
Beta Was this translation helpful? Give feedback.
-
(づ◡﹏◡)づ [llama.cpp]$ ./build/bin/llama-bench -m '/mnt/gamu/AI/textModels/gemma-3-27b-it-Q6_K.gguf' -ngl 44 -t 48 -fa 1
build: c9c64de (5431) (づ◡﹏◡)づ [llama.cpp]$ ./build/bin/llama-bench -m '/mnt/gamu/AI/textModels/gemma-3-27b-it-Q6_K.gguf' -ngl 44 -t 48 -fa 1
build: c9c64de (5431) |
Beta Was this translation helpful? Give feedback.
-
|
just to add ive been using gemma3 quite a bit its pretty usable, the model can fit fully into vram at around 30 t/s but the benchmark tool has 0 context i cant get that in actual usage |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
./build/bin/llama-bench -m '/mnt/gamu/AI/textModels/gemma-3-27b-it-q4_0.gguf' -ngl 99 -t 48 -fa 1
build: c9c64de (5431) ./build/bin/llama-bench -m '/mnt/gamu/AI/textModels/gemma-3-27b-it-q4_0.gguf' -ngl 99 -t 48 -fa 1
build: c9c64de (5431) it is possible to run llama server with the full gemma3 context size in vram, i used rocm here ./build/bin/llama-server -m '/mnt/gamu/AI/textModels/gemma-3-27b-it-q4_0.gguf' --host 0.0.0.0 --port 3344 -ngl 99 -t 48 -fa -c 131072 --cache-reuse 1 -ctk q8_0 -ctv q8_0 load_tensors: loading model tensors, this can take a while... (mmap = true) prompt eval time = 12785.13 ms / 9664 tokens ( 1.32 ms per token, 755.88 tokens per second) although this model quant level will probably perform worse ill use it for a day or so and see how it is, the 7 t/s doesnt bother me much though |
Beta Was this translation helpful? Give feedback.
-
|
Intel Arc B570
./build-vk/bin/llama-bench -m ./llama-2-7b.Q4_0.gguf -ngl 100
build: 8e186ef (5449)
./build-vk/bin/llama-bench -m ./llama-2-7b.Q4_0.gguf -ngl 100 -fa 1
build: 8e186ef (5449)
./build-vk-coop-mat/bin/llama-bench -m ./llama-2-7b.Q4_0.gguf -ngl 100
build: 8e186ef (5449)
./build-vk-coop-mat/bin/llama-bench -m ./llama-2-7b.Q4_0.gguf -ngl 100 -fa 1
build: 8e186ef (5449) |
Beta Was this translation helpful? Give feedback.
-
|
That is ... surprisingly slow, compared to my RX 580 from 2017. Especially so with the new Arc Pro B50/60 cards on the horizon using the same chip. |
Beta Was this translation helpful? Give feedback.
-
./build-sycl/bin/llama-ls-sycl-device
./build-sycl/bin/llama-bench -m ./llama-2-7b.Q4_0.gguf -ngl 100 -sm none -mg 0
build: 8e186ef (5449)
./build-sycl-f16/bin/llama-bench -m ./llama-2-7b.Q4_0.gguf -ngl 100 -sm none -mg 0
build: 8e186ef (5449) I forgot to mention in my original comment but the GPU is running on PCIe 3.0 due to my CPU (Ryzen 5 5600GT), I don't know how much that impacts the result. |
Beta Was this translation helpful? Give feedback.
-
|
I also tested it with OpenCL and somehow tg128 is much better.
./build-cl/bin/llama-bench -m ./llama-2-7b.Q4_0.gguf -ngl 100
build: 8e186ef (5449) |
Beta Was this translation helpful? Give feedback.
-
|
Man, these results are wild. Thank you for sharing them! |
Beta Was this translation helpful? Give feedback.
-
|
I was curious so I ran a Vulkan vs Cuda benchmark on an A100 GPU (80GB variant). Results are below. I was very impressed by how fast Vulkan is! Cuda is of course faster, but the Vulkan is not that far behind. ../llama.cpp/build_cuda/bin/llama-bench -m llama-2-7b.Q4_0.gguf -fa 0,1
build: d394a9a (5454) $ ../llama.cpp/build_vulkan/bin/llama-bench -m llama-2-7b.Q4_0.gguf -fa 0,1
|
Beta Was this translation helpful? Give feedback.
-
|
AMD Ryzen 5 8600G (Zen4 APU 6c/12t, 760M, RDNA3, 2x32GB DDR5-5600) OS: Debian 13 build: 9ecf3e6 (5466) |
Beta Was this translation helpful? Give feedback.
-
|
5070 Ti Vulkan llama-bench.exe -m D:\models\llama-2-7b.Q4_0.gguf -fa 0,1
build: d13d0f6 (5468) Cuda llama-bench.exe -m D:\models\llama-2-7b.Q4_0.gguf -fa 0,1
build: d13d0f6 (5468) |
Beta Was this translation helpful? Give feedback.
-
|
Intel Arc A770 (16Gb)
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 32768 | int dot: 1 | matrix cores: none
build: 259469c (5474) |
Beta Was this translation helpful? Give feedback.
-
|
Update to the Radeon RX 9070 XT numbers, with Linux 6.15.0 and newer
build: 6f180b9 (5498) Adding ROCm run using ROCm 6.4.1 (which is the first to officially support
build: cdf94a1 (5501) Got rocWMMA to build + install from latest
build: cdf94a1 (5501) |
Beta Was this translation helpful? Give feedback.
-
|
Hey, there seems to have been a fair number of performance improvements in the code. Last time I posted in this discussion was for b4646. As with last time, I will include both Vulkan and HIP for completeness sake.
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XTX (AMD proprietary driver) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 32768 | int dot: 1 | matrix cores: KHR_coopmat
Under Vulkan, it is almost 6% faster than it was a couple months ago.
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32
For HIP, a ~10% improvement since last time. Here is an interesting thing: FA under HIP is now faster than without. Vulkan with FA is still slower than without.
Meaning almost 15% faster over what HIP without FA was a few months ago, 4% faster than what HIP without FA is now, and 9% faster than what Vulkan is now. |
Beta Was this translation helpful? Give feedback.
-
|
Card: AMD Radeon PRO W5700 (
build: 4265a87 (5499) |
Beta Was this translation helpful? Give feedback.
-
|
RTX 3070 8Gb For comparison, CUDA benchmark: |
Beta Was this translation helpful? Give feedback.
and others
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
This is similar to the Apple Silicon benchmark thread, but for Vulkan! Many improvements have been made to the Vulkan backend and I think it's good to consolidate and discuss our results here.
We'll be testing the Llama 2 7B model like the other thread to keep things consistent, and use Q4_0 as it's simple to compute and small enough to fit on a 4GB GPU. You can download it here.
Instructions
Either run the commands below or download one of our Vulkan releases. If you have multiple GPUs please run the test on a single GPU using
-sm none -mg YOUR_GPU_NUMBERunless the model is too big to fit in VRAM.Share your llama-bench results along with the git hash and Vulkan info string in the comments. Feel free to try other models and compare backends, but only valid runs will be placed on the scoreboard.
If multiple entries are posted for the same device newer commits with substantial Vulkan updates are prioritized, alternatively the one with the highest tg128 score will be used. Performance may vary depending on driver, operating system, board manufacturer, etc. even if the chip is the same. For integrated graphics note that your memory speed and number of channels will greatly affect your inference speed!
Vulkan Scoreboard for Llama 2 7B, Q4_0 (no FA)
Vulkan Scoreboard for Llama 2 7B, Q4_0 (with FA)
Beta Was this translation helpful? Give feedback.
All reactions