[User] Implement Streaming LLM - Make the inference more efficient

[errorsandwarnings]

Prerequisites

Context length limit is an issue on all LLMs. The following repository and associated paper is demonstrating that keeping the 4 initial tokens will enable a infinite context length on most common LLMs without sacrificing performance or efficiency.

Code : https://github.com/mit-han-lab/streaming-llm

Paper reference inside the repo which demonstrates the attention-sink effect of LLMs and how to take advantage of it.

Current Behavior

There is a limit on context length defined mostly by pre-training. Other approaches like rope or sliding window have their pros and cons, none of them can get to a higher context length than this apporach.

[ggerganov]

Yup, I've already read the paper and the good news is that after #3228 this technique is trivial to support in llama.cpp.
It's technically already implemented by setting n_keep == 4 in main

[Dampfinchen]

They also demonstrate how memory efficient this technique is (without it, it runs OOM very early). Does that basically mean infinite context with steady and low VRAM usage? That would be revolutionary and render vectorDBs obsolete.

[nathanodle]

They also demonstrate how memory efficient this technique is (without it, it runs OOM very early). Does that basically mean infinite context with steady and low VRAM usage? That would be revolutionary and render vectorDBs obsolete.

This isn't infinite context the way you're reading it. It uses a novel modification to attention so that you can generate "infinite" output

[staviq]

If somebody can confirm if I'm understanding that paper right, I'd be grateful:

They are proposing a solution for infinite text length, not infinite context length, right ? And their observation is that the consistency of "internal state" depends heavily on first tokens, so naively keeping initial tokens and implementing sliding context window on the rest, allows the LLM to keep its sanity intact, and first tokens are "important" because appending tokens from one side, makes the first token "exist" for N iterations, second token for N-1 iterations etc, so the first tokens are "seen" by all subsequent iterations but not the other way around ?

[errorsandwarnings]

They also demonstrate how memory efficient this technique is (without it, it runs OOM very early). Does that basically mean infinite context with steady and low VRAM usage? That would be revolutionary and render vectorDBs obsolete.

This isn't infinite context the way you're reading it. It uses a novel modification to attention so that you can generate "infinite" output

Yes, it took me a while to understand and their newly posted FAQ clarifies it. I have edited the title. I believe it is a good technique to be part of this project.

[Dampfinchen]

True. This one however...

https://github.com/tomaarsen/attention_sinks

is potentially relevant. It's about the input sequence length, so the context this time. Look at that steady memory usage!

[KerfuffleV2]

It's about the input sequence length, so the context this time. Look at that steady memory usage!

The key point is the input sequence length can be very long but the context the model is considering stays constant. So you could feed it a book, have it write a book worth of content but it won't "remember" or take into account what was in the sequence 4,096 tokens ago or whatever.

[errorsandwarnings]

The VRAM usage is impressive.

[phillip-kravtsov]

I think we need n_keep=4 but also n_discard=1 here to correctly implement StreamingLLM: https://github.com/ggerganov/llama.cpp/blob/master/examples/main/main.cpp#L504.

Assuming that the llama.cpp shifting of the k-cache matches the behavior described in the paper, where the keys are rotated according to their position in the cache rather than the text (which I believe they are but not certain)

[ggerganov]

I think we need n_keep=4 but also n_discard=1 here to correctly implement StreamingLLM:

That's correct, although I don't think one really needs to shift each new token. The benefit would be marginal.

[SlyEcho]

I think the biggest impact this would give is to remove the expensive reevaluation that is currently being done but the effect is the same as --n-keep 4.

This is the RoPE handling part: modify_llama.py. I need time to understand it...

[ggerganov]

There is no longer expensive re-evaluation in main and server since #3228 was merged. Instead of re-evaluating, we are now "shifting" the KV cache which is a relatively cheap operation and in some sense is equivalent to the approach proposed in the paper. I would say we even have an advantage because we have the option to set n_discard == 8 for example which would make RoPE recalculation 1 every 8 tokens instead of on each token as it is done in StreamingLLM

[Tostino]

Just wondering, does this give us the option to choose where the sliding window begins? e.g. I have a prompt template as seen here:

<#meta#>
- Date: 2023-10-05
- Task: chat
<#system#>
You are a conversational AI having a turn based chat with a user.
<#chat#>
<#user#>
Message 1
<#bot#>
Response 1
<#user#>
Message 2
<#user_context#>
Some Context
<#bot#>
Response 2
<#user#>
Message 3
<#bot#>
Response 3

Could I anchor this portion:

<#meta#>
- Date: 2023-10-05
- Task: chat
<#system#>
You are a conversational AI having a turn based chat with a user.
<#chat#>

And have the kv cache shift only over the chat portion?

Or am I just misunderstanding things?