Implement (properly) different chat templates in main.cpp

[ngxson]

Motivation

From the day I added llama_chat_apply_template #5538 , I already started thinking about adding it into main.cpp for replacing the current -cml option. However, it is not as easy as it seems. The main reason is because main.cpp still rely on antiprompt and static prefix/postfix/infix to work with "chat".

The whole reason why antiprompt exist in the first place was because in the early era of LLMs:

• We don't have a good way to differentiate different roles (user - assistant)
• We don't have a good way to know when to stop the generation

However, a lot of things changed since then: we're now having the notion of "chat template", newer models have special tokens like <|user|> to replace Human:, most models are fine-tuned to stop generation by outputting EOS token...

For that reason, using antiprompt and static prefix/postfix/infix is no longer a viable option to add chat template into main.cpp. That force us to be a bit more creative.

Possible Implementation

• The prefix/postfix can be changed dynamically based on message role. For example:
  Chatml uses "<|im_start|>" + role + "\n" as prefix (role is dynamic based on current message); <|im_end|>\n is the postfix.
  This idea is being implemented in Refactor chat template API #6822
• Use llama_token_is_eog() to replace antiprompt. Additionally, for compatibility reason, we can translate EOG token to antiprompt, because some models output the antiprompt as a sequence of multiple tokens (newer models never do this).

Old proposal (outdated)

Possible Implementation

My idea is trying to use llama_chat_apply_template in main.cpp. This will effectively deprecate antiprompt, prompt prefix/postfix and cml options.

Format the chat on-the-go

For now, llama_chat_apply_template produce very "additive" result when a new message is added to the list.

An additive means for example if I have [msg1, msg2], then I get formatted chat msg1_msg2. When I add msg3 to the list, it must add the formatted msg3 to the end of the formatted chat, without touching the existing content, it results in msg1_msg2_msg3 in this example. A wrong result maybe msg1+++msg2_msg3

This is very important. Unlike server.cpp where we clear the KV cache and re-format a new prompt each time, main.cpp add new tokens on top of existing ones, then continues the generation until a condition is met (maybe EOS token or a stop sequence).

So, to use llama_chat_apply_template in main.cpp, a test case must be added to test all chat templates to make sure they are all "additive".

main.cpp can then keep track of a list of messages, re-apply chat template each time and take only the "added" part.

Example:

• Messages: [msg1, msg2] ==> Formatted: <user>msg1<assistant>msg2
• Messages: [msg1, msg2, msg3] ==> Formatted: <user>msg1<assistant>msg2<user>msg3 ==> Part to evaluate: <user>msg3

Manage stop sequences

While it is ideal to use stop token (for example, EOS or <|im_end|>) to stop generation, not all models support this (some models still breaks <|im_end|> into <|, im, _end, |>), so using stop token is not an option.

llama_chat_apply_template should returns the stop sequence along side with the formatted chat template ==> this is what we need to add.

[ngxson]

Superseded #6378

[MoonRide303]

I am still learning basics of llama.cpp, so maybe I am missing something, but shouldn't there also be separate parameter for main to inject system message in proper place, while using given chat template? Basically something like SYSTEM in Ollama's Modelfile.

[teleprint-me]

There are 2 models. The tokenizer and then the language model. The tokenizer manages how the input is managed and split into "tokens", I say "tokens" because they can be anything and represent anything.

The language model utilizes the tokenizer and is "conditioned" on sequencing of the input. So you could simply "inject" a "system prompt", but it wouldn't make a difference unless the language model was "conditioned" with this in mind.

You can do some really interesting things with the "initial prompt" which is typically more evident in a "completion" than a "chat completion". This is really pronounced with the Mistral models because none of them technically have a "system prompt".

[hnfong]

It seems to me that the chatml formatting code in main.cpp is mostly good, sans the anti-prompt issue. Instead of trying to reuse llama_chat_apply_template is it possible to just populate the values inp_pfx/inp_sfx/cml_pfx/cml_sfx depending on the template (and you can put the token strings in llama.cpp or even make a template.cpp to hold them all)? The change I proposed above seems simpler than trying to force main.cpp move to a different model where it holds a list of messages, which runs into the problems you mention since its a fundamentally mismatch of what main.cpp is really doing.

I imagine the change is relatively straightforward and I probably can make a patch, but if it's not the direction the project wants to go, I'll probably just hack up something myself so that I can chat with llama3 in peace :D

[ngxson]

@hnfong The reason why I propose to reuse llama_chat_apply_template is because I don't want to have 2 different APIs to manage chat templates. If we want to have prefix/postfix system, it's better to refactor the whole chat template API into something easier to maintain. Furthermore, llama_chat_apply_template is designed so that developers who use llama.cpp as a 3rd party library in their project can easily format the chat. So, it's more reasonable to also demonstrate it in main (the same way we're using it on server)

Btw, the main idea here is to rethink about this subject more seriously before actually implement it - this is to prevent introducing patches, which makes it harder for future contributors and maintainers to work on the project.

[hnfong]

We might be converging to the same idea, but anyway if you want the code to be reuseable I suppose you can refactor the llama_chat_apply_template to be able to take an existing context/list/thing/whatever and ask it to add one more message? Then the original llama_chat_apply_template can just call this function repeatedly?

FWIW, I'm happy for whatever solution that gets implemented, the existing templating code in main.cpp is really unnecessarily rigid and cumbersome and needs a makeover. (I got my quick patch working for llama3 though, so hopefully that will last [i.e. no conflicts] until this issue is resolved :)

[ngxson]

My idea is simply call llama_chat_apply_template twice: with and without the last user message. Then, I can find the diff between 2 output strings and feed it into inference. This approach will require minimal effort to maintain the chat template infrastructure, while using the extract same logic for main and server (remind: server also have the notion of "prompt cache" which works the same way)

[hanishkvc]

It seems to me that the chatml formatting code in main.cpp is mostly good, sans the anti-prompt issue. Instead of trying to reuse llama_chat_apply_template is it possible to just populate the values inp_pfx/inp_sfx/cml_pfx/cml_sfx depending on the template (and you can put the token strings in llama.cpp or even make a template.cpp to hold them all)? The change I proposed above seems simpler than trying to force main.cpp move to a different model where it holds a list of messages, which runs into the problems you mention since its a fundamentally mismatch of what main.cpp is really doing.

I imagine the change is relatively straightforward and I probably can make a patch, but if it's not the direction the project wants to go, I'll probably just hack up something myself so that I can chat with llama3 in peace :D

Have a look at #6795, it allows one to chat using the existing chat template mechanism from main. In that PR, if you look at the comments, I have attached a simple patch for adding llama3 template support and with that you can talk to llama3 using main.