Support speculative decoding in `server` example

[mscheong01]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am running the latest code. Development is very rapid so there are no tagged versions as of now.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new bug or useful enhancement to share.

Feature Description

provide speculative decoding through server example.

Motivation

Noticed this topic popped up in several comments (1, 2, 3) but it seems we haven't officially opened an issue for it. I'm creating this to provide a space for focused discussion on how we can implement this feature and actually get this started.

Possible Implementation

perhaps move speculative sampling implementation to common or sampling?

[vietanh125]

Any updates for this?

[mscheong01]

@vietanh125 Not yet, but contributions are welcome 😃

[ggerganov]

There is ongoing related work in #6828. Though I haven't had time to look in details yet

[kerthcet]

Sorry, does that means the server doesn't support speculative decoding? However, I can run it with commands like below in Kubernetes

Just a sample:

spec:
  containers:
  - args:
    - -m
    - /workspace/models/llama-2-7b.Q8_0.gguf
    - -md
    - /workspace/models/llama-2-7b.Q2_K.gguf
    - --port
    - "8080"
    - --host
    - 0.0.0.0
    - -fa
    command:
    - ./llama-server

[ggerganov]

Not yet supported

[kerthcet]

Ok so the -md doesn't work here 😀

[etafund]

Also interested in this PR. Thank you to everyone contributing to a solution here.

[theo77186]

The #6828 PR is a distinct technique that uses a lookup file to speculate tokens instead of using a draft model, there seems to have less speedup than draft-based speculative decoding.

[Hoernchen]

Support would be really nice to have because now there is the offical llama 3.2 in 1b and 3b which should be suitable for 8/70b 3.1, at least according to the offical HF notebook: https://github.com/huggingface/huggingface-llama-recipes/blob/main/assisted_decoding_8B_1B.ipynb

[gelim]

Support would be really nice to have because now there is the offical llama 3.2 in 1b and 3b which should be suitable for 8/70b 3.1, at least according to the offical HF notebook: https://github.com/huggingface/huggingface-llama-recipes/blob/main/assisted_decoding_8B_1B.ipynb

Yeah definitely. With draft: LLama-3.2-3B Q8 and model LLama-3.1-70B-Instruct (Q5_K, to fit on 2 32GB Tesla V100)

we go from 10 t/s to 30 t/s. Very impressive I'd say.

CUDA_VISIBLE_DEVICES=0,1 ./llama-speculative \
-m Meta-Llama-3.1-70B-Instruct-Q5_K_M-00001-of-00002.gguf \
-md Llama-3.2-3B-Instruct-Q8_0.gguf \
-p "// Quick-sort implementation in C (4 spaces indentation + detailed comments) and sample usage" \
 -t 4  -n 512 -c 8192 -s 8 --top_k 1 \
--draft 16 -ngl 88 -ngld 30 --temp 0

encoded   18 tokens in    0.286 seconds, speed:   62.924 t/s
decoded  514 tokens in   15.774 seconds, speed:   32.586 t/s

n_draft   = 16
n_predict = 514
n_drafted = 688
n_accept  = 470
accept    = 68.314%

draft:

llama_perf_context_print:        load time =    2505.46 ms
llama_perf_context_print: prompt eval time =    9569.42 ms /   103 tokens (   92.91 ms per token,    10.76 tokens per second)
llama_perf_context_print:        eval time =    5622.49 ms /   645 runs   (    8.72 ms per token,   114.72 tokens per second)
llama_perf_context_print:       total time =   16079.15 ms /   748 tokens

target:

llama_perf_sampler_print:    sampling time =     112.92 ms /   514 runs   (    0.22 ms per token,  4551.81 tokens per second)
llama_perf_context_print:        load time =   23527.56 ms
llama_perf_context_print: prompt eval time =    9077.52 ms /   749 tokens (   12.12 ms per token,    82.51 tokens per second)
llama_perf_context_print:        eval time =       0.00 ms /     1 runs   (    0.00 ms per token,      inf tokens per second)
llama_perf_context_print:       total time =   18584.66 ms /   750 tokens

EDIT: with LLama-3.2-1B Q8 that can go to 40 t/s

[enn-nafnlaus]

Wait, what happened? I used to run llama-server with speculative decoding with -md. I just "upgraded" and -md went away. now there's a separate program called llama-speculative, but doesn't appear to be a server. Sigh :( Guess I have to downgrade and find the version where it went away....