I'm pegging CPU (`./examples/chat.sh` works very slowly) on a 5800X3D / u22 linux, anything that can be done?

[robbintt]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

Faster responses.

Current Behavior

Used all 16 threads / 8 cores for seconds to minutes when responding to chat mode.

Environment and Context

Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.

• Physical (or virtual) hardware you are using, e.g. for Linux:

(llama.cpp) 🔥 lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  16
  On-line CPU(s) list:   0-15
Vendor ID:               AuthenticAMD
  Model name:            AMD Ryzen 7 5800X3D 8-Core Processor
    CPU family:          25
    Model:               33
    Thread(s) per core:  2
    Core(s) per socket:  8
    Socket(s):           1
    Stepping:            2
    Frequency boost:     enabled
    CPU max MHz:         4548.8281
    CPU min MHz:         2200.0000
    BogoMIPS:            6787.04
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht 
                         syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid
                          aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx
                          f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs s
                         kinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ss
                         bd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap cl
                         flushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local
                          clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushb
                         yasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes
                          vpclmulqdq rdpid overflow_recov succor smca fsrm
Virtualization features: 
  Virtualization:        AMD-V
Caches (sum of all):     
  L1d:                   256 KiB (8 instances)
  L1i:                   256 KiB (8 instances)
  L2:                    4 MiB (8 instances)
  L3:                    96 MiB (1 instance)
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-15
Vulnerabilities:         
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Not affected
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl and seccomp
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affe
                         cted
  Srbds:                 Not affected
  Tsx async abort:       Not affected

* Operating System, e.g. for Linux:
* 
Linux stardart 5.15.0-58-generic #64-Ubuntu SMP Thu Jan 5 11:43:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

* SDK version, e.g. for Linux:

(llama.cpp) 🔥 python3 --version
Python 3.10.6
(llama.cpp) 🔥 make --version
GNU Make 4.3
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
(llama.cpp) 🔥 g++ --version
g++ (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

# Failure Information (for bugs)

n/a

# Steps to Reproduce

Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.

1. Use a AMD Ryzen 7 5800X3D, 64gb ram, and  
2. install & run ./examples/chat.sh


# Failure Logs

n/a

Example environment info:

🔥 git log | head -1
commit a0c0516
🔥 lscpu | egrep "AMD|Flags"
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 7 5800X3D 8-Core Processor
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm
Virtualization:

(llama.cpp) 🔥 python3 --version
Python 3.10.6
(llama.cpp) 🔥 pip list | egrep "torch|numpy|sentencepiece"
numpy 1.24.2
sentencepiece 0.1.97
torch 2.0.0
(llama.cpp) 🔥 md5sum ./models/7B/ggml-model-q4_0.bin
f8b83a4351a2c4413aa1bb9bb995556f ./models/7B/ggml-model-q4_0.bin

perf:

(llama.cpp) 🔥 sudo perf stat ./main -m ./models/7B/ggml-model-q4_0.bin -t 16 -n 1024 -p "Please close your issue when it has been answered."
main: seed = 1680500041
llama_model_load: loading model from './models/7B/ggml-model-q4_0.bin' - please wait ...
llama_model_load: n_vocab = 32000
llama_model_load: n_ctx = 512
llama_model_load: n_embd = 4096
llama_model_load: n_mult = 256
llama_model_load: n_head = 32
llama_model_load: n_layer = 32
llama_model_load: n_rot = 128
llama_model_load: f16 = 2
llama_model_load: n_ff = 11008
llama_model_load: n_parts = 1
llama_model_load: type = 1
llama_model_load: ggml map size = 4017.70 MB
llama_model_load: ggml ctx size = 81.25 KB
llama_model_load: mem required = 5809.78 MB (+ 1026.00 MB per state)
llama_model_load: loading tensors from './models/7B/ggml-model-q4_0.bin'
llama_model_load: model size = 4017.27 MB / num tensors = 291
llama_init_from_file: kv self size = 256.00 MB

system_info: n_threads = 16 / 16 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | VSX = 0 |
sampling: temp = 0.800000, top_k = 40, top_p = 0.950000, repeat_last_n = 64, repeat_penalty = 1.100000
generate: n_ctx = 512, n_batch = 8, n_predict = 1024, n_keep = 0

Please close your issue when it has been answered.
Please do not use this category for questions that can be directed to the main support page or other sections of our website. [end of text]

llama_print_timings: load time = 7921.97 ms
llama_print_timings: sample time = 15.50 ms / 26 runs ( 0.60 ms per run)
llama_print_timings: prompt eval time = 8986.05 ms / 11 tokens ( 816.91 ms per token)
llama_print_timings: eval time = 92976.12 ms / 25 runs ( 3719.04 ms per run)
llama_print_timings: total time = 102616.54 ms

Performance counter stats for './main -m ./models/7B/ggml-model-q4_0.bin -t 16 -n 1024 -p Please close your issue when it has been answered.':

  1,461,040.62 msec task-clock                #   14.211 CPUs utilized          
     3,515,608      context-switches          #    2.406 K/sec                  
         1,639      cpu-migrations            #    1.122 /sec                   
       590,140      page-faults               #  403.918 /sec                   

6,088,040,635,313 cycles # 4.167 GHz (83.32%)
6,073,514,788 stalled-cycles-frontend # 0.10% frontend cycles idle (83.36%)
9,381,196,263 stalled-cycles-backend # 0.15% backend cycles idle (83.34%)
13,159,707,650,067 instructions # 2.16 insn per cycle
# 0.00 stalled cycles per insn (83.31%)
3,292,654,196,851 branches # 2.254 G/sec (83.35%)
187,524,242 branch-misses # 0.01% of all branches (83.35%)

 102.808235293 seconds time elapsed

1439.049630000 seconds user
  17.789593000 seconds sys

[MillionthOdin16]

To be honest I'm trying to help you, but the effort put into this issue to explain what's going on is so low...

You took the time to dump the system information, but I Don't think you actually wrote more than a couple words. There's different types of performance issues going on right now.

Is the slow down straight from the start, or does it get worse over time? And around how many tokens per second are you getting?

edit:
Bro, and you're a software dev... 🙄

[robbintt]

I haven't spent any time on this project but can if it would be useful. I am not even sure this project is intended for non Mac M2 hardware.

…

On Tue, Apr 4, 2023 at 23:54 MillionthOdin16 ***@***.***> wrote: To be honest I'm trying to help you, but the effort put into this issue to explain what's going on is so low... You took the time to dump the system information, but I Don't think you actually wrote more than a couple words. There's different types of performance issues going on right now. Is the slow down straight from the start, or does it get worse over time? And around how many tokens per second are you getting? edit: Bro, and you're a software dev... 🙄 — Reply to this email directly, view it on GitHub <#735 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOZDW2MY2V42V5XLDJ6FPDW7UJJNANCNFSM6AAAAAAWQ2KWZA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[0xdevalias]

It may not be relevant to this issue (hard to tell based on quick skim of the above), but updated context from #603 sounds like things there may have been fixed; which might potentially help here too?

---

Thank you for this hard work - I missed this regression because I rarely run generations with more than a few tens of tokens. The problem is that the transpose operation for the V matrix is very slow and becomes slower and slower with every new token added.

I think I have provided a fix here: #775

Tested only on M1 so far

Originally posted by @ggerganov in #603 (comment)

---

#775 was merged pretty quickly, cyyynthia if it's not too much trouble, could you update your pretty graphs with that?

@sw The graphs aren't super pretty this time because I didn't take the time to properly close everything and had a bunch of things open in background while the test was running 😅

That being said, the regression appears to be gone. 🎉 Here's the graphs and the raw CSV:

token_times.csv

Originally posted by @cyyynthia in #603 (comment)

---

Well I guess that settles it. The Great Wizard Georgi has saved the day! Thanks to @cyyynthia and @KASR for putting in the hard work of tracking this down.

I have opened #790 to track the discrepancy in the different partial times vs total time. I think this issue could be closed. Thanks everyone.

Originally posted by @sw in #603 (comment)

[kiratp]

Try setting threads to the physical core count, not the thread count - -t 8

ggml/llama.cpp is memory bandwidth bound - there is a lot of open discussion about this in the issues.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.