llama.android: add field formatChat to control whether to parse special tokens when send message

[codezjx]

Fix issue #11264
If the user uses the chat template to send a message, the message needs to be tokenized with parse_special = true in common_tokenize() method. Add field formatChat to control whether to parse special tokens when send message in LLamaAndroid.send(). The app can set formatChat according to their actual situation.

Sample Code:

val msg = "<|im_start|>system\n" +
    "You are a helpful AI assistant<|im_end|>\n" +
    "<|im_start|>user\n" +
    "$text<|im_end|>\n" +
    "<|im_start|>assistant\n"
viewModelScope.launch {
    // Explicitly set formatChat = true
    llamaAndroid.send(msg, true)
        .catch {
            Log.e(tag, "send() failed", it)
            messages += it.message!!
        }
        .collect { messages = messages.dropLast(1) + (messages.last() + it) }
}

Before

After

[ggerganov]

For this basic example this is OK, but keep in mind that applying parse_special = true to the full context is susceptible to special token injection from the user. The correct way is to use llama_chat_apply_template.

[codezjx]

@ggerganov Thanks for your reminder!

[ggerganov]

Yup, I actually initially incorrectly thought that we handle this special token injection in the existing chat examples, but we don't. It's something to fix in the future.

[codezjx]

@ggerganov Is the special token injection you just mentioned similar to the following behavior? Tested llama-cli in conversation mode and the user's input special token <|endoftext|> is encoded as 0. It seems that the user's input content needs to be tokenized separately (applying parse_special = false).🤔

waiting for user input

> special token <|endoftext|> injection test
buffer: 'special token <|endoftext|> injection test
'
formatted: '
<|im_start|>user
special token <|endoftext|> injection test
<|im_end|>
<|im_start|>assistant
'
input tokens: [ '':198, '<|im_start|>':1, 'user':4093, '':198, 'special':19159, ' token':9624, ' ':216, '<|endoftext|>':0, ' injection':12852, ' test':1028, '':198, '<|im_end|>':2, '':198, '<|im_start|>':1, 'ass':520, 'istant':9531, '':198 ]