Skip to content

Commit 9ba8fb6

Browse files
committed
convert_hf_to_gguf: rwkv tokenizer: Don't escape sequences manually
Signed-off-by: Molly Sophia <mollysophia379@gmail.com>
1 parent bcf29ef commit 9ba8fb6

File tree

1 file changed

+1
-3
lines changed

1 file changed

+1
-3
lines changed

convert_hf_to_gguf.py

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2723,9 +2723,7 @@ def set_vocab(self):
27232723
token = token.encode("utf-8") if isinstance(token, str) else token
27242724
assert isinstance(token, bytes)
27252725
assert len(token) == token_len
2726-
token_text: str = ""
2727-
for b in token:
2728-
token_text += f"\\x{b:02x}"
2726+
token_text: str = str(token)[2:-1]
27292727
tokens.append(token_text.encode("utf-8"))
27302728
toktypes.append(gguf.TokenType.NORMAL)
27312729
remainder = vocab_size - len(tokens)

0 commit comments

Comments
 (0)