Can't set n_gqa with server

[rodrigo-pedro]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

Being able to run the server with --n_gqa 8

Current Behavior

 warnings.warn(
usage: __main__.py [-h] [--model MODEL]
                   [--model_alias MODEL_ALIAS] [--n_ctx N_CTX]
                   [--n_gpu_layers N_GPU_LAYERS]
                   [--tensor_split TENSOR_SPLIT]
                   [--rope_freq_base ROPE_FREQ_BASE]
                   [--rope_freq_scale ROPE_FREQ_SCALE]
                   [--seed SEED] [--n_batch N_BATCH]
                   [--n_threads N_THREADS] [--f16_kv F16_KV]
                   [--use_mlock USE_MLOCK] [--use_mmap USE_MMAP]
                   [--embedding EMBEDDING] [--low_vram LOW_VRAM]
                   [--last_n_tokens_size LAST_N_TOKENS_SIZE]
                   [--logits_all LOGITS_ALL] [--cache CACHE]
                   [--cache_type CACHE_TYPE]
                   [--cache_size CACHE_SIZE]
                   [--vocab_only VOCAB_ONLY] [--verbose VERBOSE]
                   [--host HOST] [--port PORT]
                   [--interrupt_requests INTERRUPT_REQUESTS]
                   [--n_gqa N_GQA] [--rms_norm_eps RMS_NORM_EPS]
__main__.py: error: argument --n_gqa: invalid Optional value: '8'

[skrbnv]

There's an issue with how main.py processes argument type, it can't deal with Optional[int]
As a quick fix you can locate where package is stored and change Optional[int] to int

It's in app.py, Settings
n_gqa: Optional[int] = Field( default=None, description="TEMPORARY: Set to 8 for Llama2 70B", )

[sai-samarth]

Can confirm that by changing it to:

n_gqa: int = Field( default=None, description="TEMPORARY: Set to 8 for Llama2 70B", )

works

[maxchen-000]

you can also set the environment variable:
export N_GQA=8

[abetlen]

@saisamarthtaluri @skrbnv thank you for pointing this out, I'll push the fix shortly!

[gjmulder]

Duplicate of #524

[JasonCaoJR]

In the file "llama_cpp/server/app.py", if change it to

n_gqa: Union[int, None] = Field(

will work. Anyone could make a change?

[gaby]

@gjmulder @abetlen Any updates on this? I can't get LlaMA2-7b to work. It may be related to this.

lama.cpp: loading model from /usr/src/app/weights/Facebook-LLaMA2-7B.bin
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 32000
llama_model_load_internal: n_ctx      = 512
llama_model_load_internal: n_embd     = 4096
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 32
llama_model_load_internal: n_head_kv  = 4
llama_model_load_internal: n_layer    = 32
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: n_gqa      = 8
llama_model_load_internal: rnorm_eps  = 5.0e-06
llama_model_load_internal: n_ff       = 11008
llama_model_load_internal: freq_base  = 10000.0
llama_model_load_internal: freq_scale = 1
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: model size = 7B
llama_model_load_internal: ggml ctx size =    0.08 MB
error loading model: llama.cpp: tensor 'layers.0.attention.wk.weight' has wrong shape; expected  4096 x   512, got  4096 x  4096

[abetlen]

With the transition to gguf the n_gqa value is set automatically and has been removed from the llama.cpp api so I'll close this issue.