Tokenization is not equal to Meta's tokenization.

[viniciusarruda]

I'm comparing the tokenization between original Meta repo and llama.cpp with LLaMA (also had same issue with LLaMA v2).

For example, tokenizing the prompt "Hello world" and " Hello world" gives the following:

For prompt "Hello world":
llama.cpp tokenizer: [10994, 3186]
Meta tokenizer: [15043, 3186]

For prompt " Hello world":
llama.cpp tokenizer: [15043, 3186]
Meta tokenizer: [29871, 15043, 3186]

Exploring the tokens, doing the detokenization, I got:

For tokens "[10994, 3186]":
llama.cpp tokenizer: |b'Hello world'|
Meta tokenizer: |Hello world|

For tokens "[15043, 3186]":
llama.cpp tokenizer: |b' Hello world'|
Meta tokenizer: |Hello world|

For tokens "[29871, 15043, 3186]":
llama.cpp tokenizer: |b' Hello world'|
Meta tokenizer: | Hello world|

*Adding | to ease visualization.

Exploring each token above with the id_to_piece functionality:

Looking the id_to_piece for llama.cpp:
id 10994 |b'Hello'|
id 3186 |b' world'|
id 15043 |b' Hello'|
id 29871 |b' '|

Looking the id_to_piece for Meta:
id 10994 |Hello|
id 3186 |▁world|
id 15043 |▁Hello|
id 29871 |▁|

*Adding | to ease visualization.
Note, the 29871 token is not the underline character but "\u2581" (See more about this here).

But, using the detokenizer in each id individually:

Using the llama.cpp detokenizer:
id 10994 |b'Hello'|
id 3186 |b' world'|
id 15043 |b' Hello'|
id 29871 |b' '|

Using the Meta detokenizer:
id 10994 |Hello|
id 3186 |world|
id 15043 |Hello|
id 29871 ||

The code used to produce this results can be seen here.
Use this file for the Meta tokenizer.
The model ggml-model-f16.bin is the 7B LLaMA model after using the convert.py script as mentioned here.

[goerch]

There are known problems with the tokenizer,

[jxy]

because Tokenizer.tokenize() always add a space in front of the string passed in.

29871 is the space

[jxy]

just to explain what I meant by "space", see here: https://github.com/google/sentencepiece/blob/master/README.md#whitespace-is-treated-as-a-basic-symbol

[goerch]

because Tokenizer.tokenize() always add a space in front of the string passed in.

I don't think it does that, because I just tested it here. But may be this happens elsewhere.

[goerch]

For example, tokenizing the prompt "Hello world" and " Hello world" gives the following:

For prompt "Hello world":
llama.cpp tokenizer: [10994, 3186]
Meta tokenizer: [15043, 3186]
For prompt " Hello world":
llama.cpp tokenizer: [15043, 3186]
Meta tokenizer: [29871, 15043, 3186]

Running the tests I see the Meta tokens now. But I surely need guidance on how to integrate the necessary changes into llama.cpp .

BTW: I'd expect a couple of more problems for other tests.

[goerch]

@viniciusarruda and others ; nice find, indeed! Now that I've learned a bit about the tokenizer: this kind of problem could not only occur with llama but with derived models using different tokenizers (w.r.t. vocabulary for example) ? But the common theme here is sentencepiece compatibility? You made a big step with your notebook (which I used for testing already, thanks!), could you advise us regarding our future testing strategy?

[callMeMakerRen]

i am curious, from openai repo, How_to_count_tokens_with_tiktoken,it says Encodings specify how text is converted into tokens. Different models use different encodings,so i wonder if we are on the right way? did we only need to match the meta?

[henk717]

I'd like to add to this discussion that the real llama tokenizer eats spaces at the beginning making it very bad for continuous generation. Llamacpp's tokenizer was previously tested to give the same results as the HF tokenizer with our workaround. So for continuous generation the behavior might be better. If this is changed in the code it would be nice if this would be an optional change since if the tokenizer begins eating spaces this breaks use cases.

The tokenizer eating spaces is only desirable for applications with a seperate output window.

[goerch]

I'd like to add to this discussion that the real llama tokenizer eats spaces at the beginning making it very bad for continuous generation. Llamacpp's tokenizer was previously tested to give the same results as the HF tokenizer with our workaround.

Do you have a possible test scenario for the proposed PR in mind? I don't know how to check that otherwise.

[henk717]

Yes, and the upstream tokenizer workarounds are not perfect so I will give you a method to test the ideal scenario.
Our UI strips a space at the end of a generation because most models can not handle it well if its already there because their tokens often have spaces in front of them, so we let the model decide if there should be a space or not.

So imagine a sentence that does not end with a space such as "This is an example." in a UI that lets users add to that text. The correct scenario would be this generation " It works very well." which combined forms "This is an example. This works very well."

The original Llama tokenizer at least on HF doesn't do this because they trained sentencepiece in a mode where it swallows the first space. So you'd get this "It works very well." resulting in "This is an example.It works very well".

Likewise you also don't want to always insert the space if you don't have to, "My favorite sport is foot" should auto complete to "ball and I play it every week" not " ball and I play it every week".

Adhering to that behavior would make it usable for continuous generation use cases because then the text flows naturally and the tokenizer picks the most appropriate token for the end result, rather than never picking a space or always picking a space.

On the huggingface side we force the tokenizers hand, we insert a random token such as a comma at the front of the input, and then after its tokenized we remove the first fake token which results in it doing a proper continuous generation. The behavior we typically observed from Llamacpp is very similar to that workaround. And we originally implemented workarounds like this because Llamacpp did a better job and its helped us in the past to identify things were wrong.

[goerch]

Ad 1.

.\build\bin\Release\main.exe -m models\llama-7B\ggml-model-q4_0.bin -p "This is an example." --temp 0 --color

With main:

 This is an example. It's a bit of a mess, but it shows how to use the new features in the latest version of the library.
The first thing you need to do is create a new project and add the library to it. You can find out more about this here.

With PR:

 This is an example. It's a bit of a mess, but it shows how to use the new features in the latest version of the library.
The first thing you need to do is create a new project and add the library to it. You can find out more about this here.
[...]

Ad 2.

.\build\bin\Release\main.exe -m models\llama-7B\ggml-model-q4_0.bin -p "My favorite sport is foot" --temp 0 --color

With main:

 My favorite sport is foot ball. I like to play it and watch it on TV.
I like to play football, too. It's my favourite sport. [end of text]

With PR:

 My favorite sport is foot ball. I like to play it and watch it on TV.
I like to play football, too. It's my favourite sport. [end of text]