Command-R GGUF conversion no longer working

[candre23]

As recently as a few days ago, Command-R (and presumably R+) could be converted with convert-hf-to-gguf.py. I double checked and conversion completes successfully in b2751. However, with the recent changes to accommodate Llama3, Command-R compatibility has been broken. Trying to convert today with b2777 I get

raise NotImplementedError("BPE pre-tokenizer was not recognized - update get_vocab_base_pre()")
NotImplementedError: BPE pre-tokenizer was not recognized - update get_vocab_base_pre()

I know that L3 required a new tokenizer provided by meta to facilitate proper conversion. Do we require something new from cohere, or is this something that can be fixed internally?

[drummerv]

I had the same issue and the same error! Had to rollback to before Llama 3 to get it to work.

[dranger003]

This patch appears to get it working again.

diff --git a/convert-hf-to-gguf.py b/convert-hf-to-gguf.py
index 2f146d73..e8cc040f 100755
--- a/convert-hf-to-gguf.py
+++ b/convert-hf-to-gguf.py
@@ -306,6 +306,8 @@ class Model(ABC):
         if chkhsh == "3ce83efda5659b07b1ad37ca97ca5797ea4285d9b9ab0dc679e4a720c9da7454":
             # ref: https://huggingface.co/openai-community/gpt2
             res = "gpt-2"
+        if chkhsh == "9c2227e4dd922002fb81bde4fc02b0483ca4f12911410dee2255e4987644e3f8":
+            res = "gpt-2"

         if res is None:
             print("\n")

[dranger003]

There seems to be an issue using convert-hf-to-gguf-update.py and the downloaded tokenizer.json file does not contain JSON but rather contains this:

version https://git-lfs.github.com/spec/v1
oid sha256:0af6e6fe50ce1bb5611b103482de6bac000c82e06898138d57f35af121aec772
size 12777406

This is after adding this to convert-hf-to-gguf-update.py:

{ "name": "command-r",      "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/CohereForAI/c4ai-command-r-v01", },

[candre23]

I downloaded the model manually earlier today and the tokenizer.json is definitely a real tokenizer. The file size mentioned in the placeholder that you got matches the actual size of what I got from cohere's HF repo (~12.1mb).

[sealad886]

There seems to be an issue using convert-hf-to-gguf-update.py and the downloaded tokenizer.json file does not contain JSON but rather contains this:

version https://git-lfs.github.com/spec/v1
oid sha256:0af6e6fe50ce1bb5611b103482de6bac000c82e06898138d57f35af121aec772
size 12777406

This is after adding this to convert-hf-to-gguf-update.py:

{ "name": "command-r",      "tokt": TOKENIZER_TYPE.BPE, "repo": "https://huggingface.co/CohereForAI/c4ai-command-r-v01", },

This is caused by Git LFS (for more details). To fix this, download the tokenizer.json file directly from Huggingface and replace the file that was previously downloaded:

tokenizer.json for command-r
--or--
tokenizer.json for command-r-plus

[sealad886]

As @candre23 noted, this also affects command-r-plus...and every model of a specific type (BPE pre-tokenizer? I'm not super familiar with that).
It's a non-trivial but straightforward way to add back model types to convert-hf-to-gguf.py. Follow the instructions in this pull request. Admittedly, I got a bit lost trying to figure out what regex to use for the command-r-plus pre-tokenizer type (it appears to me that Cohere have made this one unified type?).
I also might have royallly screwed up my source files accidentally. I was too tired when I started looking at this, and by the end couldn't put one thought in front of another...

[sealad886]

As @candre23 noted, this also affects command-r-plus...and every model of a specific type (BPE pre-tokenizer? I'm not super familiar with that). It's a non-trivial but straightforward way to add back model types to convert-hf-to-gguf.py. Follow the instructions in this pull request. Admittedly, I got a bit lost trying to figure out what regex to use for the command-r-plus pre-tokenizer type (it appears to me that Cohere have made this one unified type?). I also might have royallly screwed up my source files accidentally. I was too tired when I started looking at this, and by the end couldn't put one thought in front of another...

Got this working myself about the same time I saw your PR, @dranger003. It does appear this is fixed by #7033 so let's get that merged asap. :)