Support jinja extra template kwargs (Qwen3 enable_thinking feature), from command line and from client

[matteoserva]

This PR implements support for setting additional jinja parameters.
An example of this is enable_thinking in the Qwen3 models template.

Main features:

• Setting jinja variables for command line using --chat_template_kwargs or the environment variable
• Setting variables per request in the OAI compatible api using the chat_template_kwargs parameter
• Compatibility with the VLLM API

Notice

• As per server: add --reasoning-budget 0 to disable thinking (incl. qwen3 w/ enable_thinking:false) #13771 the preferred way for disabling thinking with a command line argument is now --reasoning-budget 0. The command line setting can be overridden anyway by passing the chat_template_kwargs during the request to the OAI compatible API
• There is ongoing discussion to support setting the reasoning budget per request in Feature Request: add per-request "reasoning" options in llama-server #13272. This would allow to completely disable thinking by setting the budget to 0

Other info

The official template is still only partially compatible. I modified it to use only supported features.
It's here: https://pastebin.com/16ZpCLHk https://pastebin.com/GGuTbFRc
And should be loaded with llama-server --jinja --chat-template-file {template_file}

It fixes #13160 and #13189

Test it with:

• enable_thinking=false. Expected: {"prompt":"\n<|im_start|>user\nGive me a short introduction to large language models.<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n"}

curl http://localhost:8080/apply-template -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-8B",
  "messages": [
    {"role": "user", "content": "Give me a short introduction to large language models."}
  ],
  "temperature": 0.7,
  "top_p": 0.8,
  "top_k": 20,
  "max_tokens": 8192,
  "presence_penalty": 1.5,
  "chat_template_kwargs": {"enable_thinking": false}
}'

• enable_thinking=true

curl http://localhost:8080/apply-template -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-8B",
  "messages": [
    {"role": "user", "content": "Give me a short introduction to large language models."}
  ],
  "temperature": 0.7,
  "top_p": 0.8,
  "top_k": 20,
  "max_tokens": 8192,
  "presence_penalty": 1.5,
  "chat_template_kwargs": {"enable_thinking": true}
}'

• enable_thinking undefined

curl http://localhost:8080/apply-template -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen3-8B",
  "messages": [
    {"role": "user", "content": "Give me a short introduction to large language models."}
  ],
  "temperature": 0.7,
  "top_p": 0.8,
  "top_k": 20,
  "max_tokens": 8192,
  "presence_penalty": 1.5
}'

[rhjdvsgsgks]

can you add chat_template_kwargs to cli argument as well?

[matteoserva]

can you add chat_template_kwargs to cli argument as well?

I added it. I tested it using updated command (You might want to check the escaping of the double quotes):
--chat_template_kwargs "{\"enable_thinking\":false}" --jinja --chat-template-file qwen/qwen3_template.txt

[neolee]

Very useful for Qwen3 series. +1 for this feature!

[xiaomi102]

Cannot work with the --chat_template_kwargs option from CLI: error: invalid argument: --chat-template-kwargs
git clone --depth=1 -b enable_thinking https://github.com/matteoserva/llama.cpp

[celsowm]

@ggerganov is there any reason why this PR has not been accepted and merged yet?

[matteoserva]

Cannot work with the --chat_template_kwargs option from CLI: error: invalid argument: --chat-template-kwargs git clone

This PR is implemented only for llama-server and its webui.

llama-cli has unresolved bugs that prevent me from enabling this feature.

[xiaomi102]

Cannot work with the --chat_template_kwargs option from CLI: error: invalid argument: --chat-template-kwargs git clone

This PR is implemented only for llama-server and its webui.

llama-cli has unresolved bugs that prevent me from enabling this feature.

Hope you'll integrate it for the CLI environment soon, thanks!

[matteoserva]

Hope you'll integrate it for the CLI environment soon, thanks!

I'll open a new PR soon for llama-cli. The code is ready but it's blocked by #13402 and #13404

[celsowm]

would be nice a enable_thinking checkbox or something like that on llama cpp webui too

[strawberrymelonpanda]

@ggerganov is there any reason why this PR has not been accepted and merged yet?

@celsowm Lack of eyes on this area would be my guess.

With 438 open PRs (many obsolete), I've kind of come to accept I'll need to pull in some PRs of interest to me when building.

[neolee]

@ggerganov is there any reason why this PR has not been accepted and merged yet?

@celsowm Lack of eyes on this area would be my guess.

With 438 open PRs (many obsolete), I've kind of come to accept I'll need to pull in some PRs of interest to me when building.

vLLM and SGLang have got this feature the first day Qwen3 released. At the same time many useful enhancement and fix PRs become obsolete just because of delay on merging here in llama.cpp community. Really sad about that.

[Neath]

This is so necessary when dealing with Qwen3! Can't wait to see this merged and be able to use the latest version with this <3

[ochafik]

• While being able to pass kwargs will be very useful in general (🎉), I think for the particular case of enable_thinking we will probably want to special case it, since there's a few thinking models around, some of which force the thinking tag open (a common --disable-thinking flag could close their tags, and set the enable_thinking: false for Qwen3).

Why not accept this PR to get the general "pass kwargs to chat template" feature at first, then implement the enable_thinking flag using this feature?

@neolee I think disabling thoughts should be handled more manually, the enable_thinking extra arg works for Qwen3 only, sent out #13771 for review to handle the other supported thinking models.

I added the support also to common_chat_params_init_generic() in the latest commit. I don't know if it makes sense for the other specialized init functions.

@matteoserva Thanks! I think each and every code path should pass the extra context variables if we want to support generic use cases (although as I just noted in a review comment, we should be weary of requiring users to set custom key-values for features that would be better handled in a generic & unified way across templates & models; ideally most users won't need to inspect any templates, and the ones who do can trivially create their own template overrides anyway).

[matteoserva]

Just to make explicit, the added value of this PR is that:

• You can set variables per request instead of restarting the server with a different template
• Compatibility with the VLLM api, so the same code works for both engines

[aviallon]

Thanks for this PR @matteoserva, I'm cherry-picking it for my own use :)

[ochafik]

Looks good to me after a few nits are addressed.

@ngxson would you mind giving it a final look and approving it?

[ochafik]

Took the liberty to push an update in your branch with support for propagating the extra context for all the chat formats with slightly simpler code.

Could you please update the PR description mentioning the new alternative way to disable the thinking globally (#13771, and discussions on upcoming per-request mechanism for it: #13272), and adding a link to the VLLM API mentioning the new param has the same syntax (I only realized that from your last comment :-))

[matteoserva]

I applied the requested changes but the latest rebase has been rough. Please review more carefully to make sure I didn't do any mistake (sorry).

[qinxuye]

We do need this PR, maintainers please review this.