Feature Request: add per-request "reasoning" options in llama-server

[ngxson]

Feature Description

As reasoning models are becoming mainstream, we start to see some pattern:

• Most models use <think>, <reasoning>, etc, basically a set of known tokens now
• The "reasoning budget" can technically be supported by any models, not just Qwen, by keeping track of number of tokens between <think> and </think>
• "no think" is just a reasoning budget == 0

So I'm thinking about accepting an object like this for each request:

"reasoning": {
    "budget": -1, // number of reasoning tokens budget
                     default: -1 (inf) ; 0 for no think
    "format": "", // equivalent of --reasoning-format
                     if set to "deepseek", reasoning will be returned in "message.reasoning_content"
                     if set to "hide", it will be completely hidden
                     default: "none", return the reasoning with the message as normal
}

The reasoning format "hide" can be implemented via #13214 ; the "deepseek" format current only supported for non-stream, but I think we can modify a bit to support this.

For the budget, we don't yet have the logic to handle it.

[GreenCappuccino]

Another interesting option, maybe expose reasoning_effort as a Jinja templating variable? Could be used in Qwen3 where low could fill the <think> block ahead of time, and be OpenAI compatible.

[ngxson]

Yes we could but I think that will be a one-off solution, i.e. if tomorrow someone else drop another reasoning model, that won't work anymore.

A more generic solution will be much better in the long term.

[Interpause]

Assuming most reasoning models in the future work by having a thinking section, the reasoning effort can be controlled by live-adjusting the logit bias for the thinking termination token (i.e., </think> for qwen). That should be a pretty genetic way to do it.

Inspiration is from https://github.com/and270/thinking_effort_processor who impmemented that.

[nattliang]

According to the Qwen 3 paper, they manually added a custom stop thinking instruction to the output after the thinking budget was reached. It could potentially be incorporated into the prompt template.

[ochafik]

Exciting idea! API-wise you might want to align w/ Anthropic's API: thinking.{budget_tokens,type} (edit: although their extended thinking seems disabled by default, while llama.cpp might want to default to each model's behaviour?)

[ngxson]

@ochafik I think thinking.budget_tokens can be useful for us to adopt, as in the future we can have different types of budgets, like by number of words, characters or time budget.

For thinking.type, I don't think we need it as it is too model-dependent.