server: Completion of pre-tokenized prompt is broken

[shibe2]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code: cafcd4f
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Expected Behavior

According to documentation: https://github.com/ggerganov/llama.cpp/blob/cafcd4f89500b8afef722cdb08088eceb8a22572/examples/server/README.md?plain=1#L117

or as an array of strings or numbers representing tokens

Current Behavior

When supplying the prompt as array of token identifiers, it instead calls split_multiprompt_task and the request hangs.

Steps to Reproduce

1. Call /tokenize with a text prompt in content.
2. Add BOS if needed.
3. Call /completion with the resulting array in prompt.

Failure Logs

all slots are idle and system prompt is empty, clear the KV cache

slot 0 is processing [task id: 2]
slot unavailable

print_timings: prompt eval time = 0.00 ms / 0 tokens ( -nan ms per token, -nan tokens per second)
print_timings: eval time = -94366367288.92 ms / 0 runs ( -inf ms per token, -0.00 tokens per second)
print_timings: total time = -94366367288.92 ms
slot unavailable

[jxy]

It took me sometime to realize what's wrong with my server.

I originally added the prompt array support for testing prompts with specifically selected tokens, which has been quite useful, as the prompts subject no constraint of any tokenizer.

In order to support both usage, how about allowing 2-level nested prompts? For example,

{
"prompt": [
    "First prompt",
    [1, "second prompt provided in an array.", 2, 1, "What do you think?"],
    "Third prompt"
]
}

This would be compatible with the multi-prompt change already introduced, and allow for array prompts.

[jxy]

I'm baffled. I can't get the current multi-prompt to work. #4583 I'll wait for that to be fixed and introduce new behaviors. For now, using #4232 (comment) to get back the previous behavior.

[shibe2]

I would like it better if multi-prompt field was called "prompts". It can then have sub-arrays as in your example, @jxy. The format of "prompt" field can be made to match the current documentation, i.e. single prompt.

[github-actions]

This issue is stale because it has been open for 30 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.