[feature] Make space insertion optional in SentencePiece tokenizer

[shibe2]

Current Behavior

Since #2810, space is inserted into any non-empty text. This breaks multiple use cases:

• Infill Incorrect Tokenization #3503 (comment)
• Prompt size control: splitting long text into pieces and adding these pieces to the prompt until certain token limit is reached. It is desirable to know precise token count of each piece, but space insertion gets in the way here.
• Prompt formats that use added tokens. Whether a space should be inserted after a special token depends on particular model used (how it was trained/finetuned and perhaps, alignment of stars). The decision should be left to the code that composes/formats prompts.
• Yi tokenizer does not insert space.

Recently, space insertion was disabled in the case when text representation of special tokens is recognized: https://github.com/ggerganov/llama.cpp/blob/1a159553f921a9209fed8c714494e57b3649f232/llama.cpp#L6729 This works for toying with main example when escape processing is enabled, but leaves other scenarios broken. In particular, when special token identifiers are added to the prompt by client and passed to server (which is a proper way to handle added tokens), it inserts space into each piece of text between special tokens.

Space insertion was made to match original Python implementation, but that behavior is itself not optimal, as evidenced by people having to hack around it.

Proposals

Option 1

Insert space only when BOS token is also inserted. There is a clear intersection between cases where BOS and space need to be inserted: when the whole prompt is one chunk of text. All broken cases that I listed here involve splitting and recombining the prompt.

The argument bos can be optionally renamed to something like full_prompt, or inverse partial, with meaning that would encompass the behaviors controlled by it.

Option 2

Add a separate argument that controls insertion of space. It would be used by /tokenize in server and in other places.

Option 3

Add another tokenization function with options that would control many aspects of the process. Suggested by @staviq.

[ggerganov]

Option 1 might actually be a good idea

[staviq]

In particular, when special token identifiers are added to the prompt by client and passed to server (which is a proper way to handle added tokens), it inserts space into each piece of text between special tokens.

Unless there's a bug ( and I don't think there is, I tested it extensively, though it's not impossible ), it's the other way around.

Tokenizing with "special=false" will only add space at the begining of the entire input string, and special tokens will be naively tokenized as plaintext. That is in line with old tokenizer behaviour.

Tokenizing with "special=true" will not add any spaces, neither at the begining of the input string nor to the substrings between special tokens.

EDIT:

Option 1 would leave you with stray space if special tokens are enabled and add_bos is enabled.

Example:
<|im_start|>user\nHi.<|im_end|>\n

Current behavior with special=false, same as old behavior:
bos, <,|,im,_,start,|,>,user,endl,Hi,.,<,|,im,_,end,|,>,endl

Current (new) behaviour with special=true:
bos,<|im_start|>,user,endl,Hi,.,<|im_end|>,endl

Tying space prefix to add bos when special=true would leave you with:
bos, ,<|im_start|>,user,endl,Hi,.,<|im_end|>,endl

[shibe2]

@staviq I just tested it.

request: {"prompt":[1,32001,"name\nmessage",32002,"\n",32001,"name\nmessage",32002,"\n",32001,"name\n"]}
tokenized: [1,32001,1024,13,4906,32002,29871,13,32001,1024,13,4906,32002,29871,13,32001,1024,13]
decoded: "<s><|im_start|> name\nmessage<|im_end|> \n<|im_start|> name\nmessage<|im_end|> \n<|im_start|> name\n"

As you can see, there are no spaces at all in the request, but tokenizer uses tokens that start with space. That is, it inserts space into each text fragment.

[staviq]

@staviq I just tested it.

request: {"prompt":[1,32001,"name\nmessage",32002,"\n",32001,"name\nmessage",32002,"\n",32001,"name\n"]} tokenized: [1,32001,1024,13,4906,32002,29871,13,32001,1024,13,4906,32002,29871,13,32001,1024,13] decoded: "<s><|im_start|> name\nmessage<|im_end|> \n<|im_start|> name\nmessage<|im_end|> \n<|im_start|> name\n"

As you can see, there are no spaces at all in the request, but tokenizer uses tokens that start with space. That is, it inserts space into each text fragment.

Oh, I see, server has it's own implementation of tokenize:

https://github.com/ggerganov/llama.cpp/blob/4e82b2ea3fa6482915d147bc9f46e70b9ada7700/examples/server/server.cpp#L269-L278

I thought you mean new special token handling broke this, but you meant it didn't fix an already existing problem in server, correct ?

[shibe2]

Yes, the compulsory space is affecting server's behavior ever since it was introduced in #2810. Maybe we can call it a problem, but I see it as a missing feature. In many cases, inserting space is desirable, but it takes control over it away from the client.

If my option 1 is chosen, it may be that nothing needs to be changed in the server code. But it should be documented in which cases the client needs to add spaces.

If my option 2 is chosen, the server can be made to behave like in option 1 by default and have optional parameters that control insertion of spaces. Again, with appropriate documentation.

[grencez]

Spaces aren't opportunistically inserted within the text, so doing it only when add_bos==true would make sense to me. (Option 1)

A little extra info: The OP links to a hack that prepends a "🙂". Pretty smart! I've been prepending a newline, but am not sure if it will reliably yield a unique newline token for all models in all cases. If tokenizers are really unpredictable, clients that edit the context should be retokenizing at arbitrary boundaries (I think this is called "token healing"), which is also complicated by space insertion.

[shibe2]

@grencez I am also using newline (LF) for working around the space insertion. I'm curious to see a model that uses SentencePiece tokenizer and has multiple tokens that contain newline, which could make this usage of newline problematic. One of things that I also do is splitting text on token boundaries, so this will be important even when space insertion will be controllable.

[grencez]

I'm curious to see a model that uses SentencePiece tokenizer and has multiple tokens that contain newline

@shibe2 The CausalLM GGUF models on HuggingFace do this sometimes. " \n\n" becomes 2 tokens (" \n" and "\n"). Those quantizations are still experimental and seem buggy (e.g., llama_token_nl() differs from how "\n" is tokenized), so don't jump to any conclusions just yet.

[shibe2]

• Adding BOS is optional after Respect tokenizer.ggml.add_bos_token value when tokenizing #4040.
• Adding initial space may become optional in the Python implementation.

Given that, BOS and initial space no longer seem to be nicely coupled. Therefore, I prefer my option 2 – adding a new argument to tokenization functions.

[KerfuffleV2]

Something definitely should happen with this. The current behavior destroys the quality of some models. See #4081 (comment)

The way we token also clearly differs from the official Python tokenizing stuff. (Example in the comment I linked.)

[staviq]

It seems the discussion is happening in parallel, here and in #4081

I looks to me like a single llama_tokenize function might not be enough to solve this, because it seems like for every possible solution so far, there is a mutually exclusive use case.

So my proposal is to add llama_tokenize_advanced ( or something similar ), and I think we might want to use single bitfiled parameter for "settings" ( ORable enum )

This way, we won't break llama.cpp bindings when adding function arguments ( we/I did accidentally break llama-cpp-python by adding special before ), and we would be able to modify and add functionality to the tokenizer, without breaking compatibility in the future.

[KerfuffleV2]

So my proposal is to add llama_tokenize_advanced

How about doing that but make it take a struct with options? Sort of like how the quantize stuff has struct llama_model_quantize_params. Then you can basically add an unlimited amount of options to it without making the parameter passing insane or breaking the API whenever a new option gets added.

[staviq]

I was about to say that, but I thought it would be to radical of a change

But this is a good idea, Vulkan for example uses that to solve it's problem of ungodly amount of parameters in function calls.