Token generation is extremely slow when using 13B models on an M1 Pro with llama.cpp, but it runs at a fine speed with Dalai (which uses an older version of llama.cpp)

[serovar]

Expected Behavior

I can load a 13B model and generate text with it with decent token generation speed with a M1 Pro CPU (16 GB RAM).

Current Behavior

When I load a 13B model with llama.cpp (like Alpaca 13B or other models based on it) and I try to generate some text, every token generation needs several seconds, to the point that these models are not usable for how unbearably slow they are. But they works with reasonable speed using Dalai, that uses an older version of llama.cpp

Environment and Context

MacBook Pro with M1 Pro, 16 GB RAM, macOS Ventura 13.3.

Python 3.9.16

GNU Make 3.81

Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: arm64-apple-darwin22.4.0
Thread model: posix

If you need some kind of log or other informations, I will post everything you need. Thanks in advance.

[j-f1]

A couple of things to check:

• Are you building in release mode? Debug mode would be significantly slower.
• Which weight type are you using? f16 weights may be slower to run than q4_0 weights.

[serovar]

I followed the instructions in the repo (simply git clone the repo, cd to the folder and make, so I suppose by default it builds in release mode).

I tried with f16 (that I understands are the q4_1 ones) and q4_0 with similar results.

I can add that if I load a basic command like ./main -m ./models/vicuna-13B/ggml-vicuna-13b-4bit.bin -n 128 the generation speed is fine, but when using the example scripts (obviously with the correct model path) it becomes unbearably slow.

Video of ./main -m ./models/vicuna-13B/ggml-vicuna-13b-4bit.bin -n 128 :

CleanShot.2023-04-04.at.20.05.20.mp4

Video of the chat-13B example script:

CleanShot.2023-04-04.at.20.07.29.mp4

[j-f1]

Hm, I don’t see a chat-13B-vicuna.sh example in the examples folder. Are you sure you’re filing this against the right repo?

[serovar]

It’s the chat-13B.sh but with the path for the vicuna model, I get the exact same results using the default chat-13B.sh with the standard alpaca 13B model (and with every other example script in that folder).

[x02Sylvie]

Probably relevant, #603

[j-f1]

My guess is that one or more of the additional options the script is passing to ./main is causing the slowdown — if you start by removing all of them and then adding them back one at a time you should be able to track down which one is causing the slowdown.

[cyyynthia]

The older version used by Dalai (https://github.com/candywrap/llama.cpp) doesn't include the changes pointed out in #603 which appear to have caused significant performance regression. My assumption is that it's related to what we're investigating over there.

[KASR]

Might be the same as issue as #735 and #677 and indeed probably related to #603

[serovar]

My guess is that one or more of the additional options the script is passing to ./main is causing the slowdown — if you start by removing all of them and then adding them back one at a time you should be able to track down which one is causing the slowdown.

I tried changing and removing the additional options without results. Moreover the strangest thing is that now even the simple ./main -m ./models/vicuna-13B/ggml-vicuna-13b-4bit.bin -n 128 command leads to slow token generation (I did not change anything at all).

CleanShot.2023-04-04.at.23.56.29.mp4

[j-f1]

I think we should close this out for now since it seems like the performance regression discussion is happening in #603 / #677 / #735

[MillionthOdin16]

I don't know that this is specifically the issue that I describe in #603. His behavior is different than mine, and might be related to memory and swap issues. I've seen problems for some users since the mmap() update, and what he's describing sounds more similar to one of those where performance plummets to unbearable levels straight from the start.

I didn't even see this issue because it was closed before I got a chance to. The only reason I saw it was because it was referenced in my issue.

@serovar can you look at your disk utilization as it's processing?

[serovar]

Here:

CleanShot.2023-04-05.at.12.05.58.mp4

[MillionthOdin16]

Okay, thanks. What's your RAM usage look like? Do you have spare ram or is it all allocated?

edit:
I'm not sure about the technical details of what's going on, but it looks like you might be using swap because your low on ram. To this point I've only seen it confirmed on Windows, but it might be related to the case where end mmap causes poor performance for users. Someone with more experience with the recent updates can probably help narrow it down.

But in relation to issue 603, this issue is different and I think it only started happening in the last few days for users. The reason you don't have the issue with dalai is because it doesn't have some of the more recent updates updates from this repo.