Implement caching for evaluated prompts

[abetlen]

The goal of this feature is to reduce latency for repeated calls to the chat_completion api by saving the kv_cache keyed by the prompt tokens.

The basic version of this is to simply save the kv_state after the prompt is generated.

Additionally we should investigate if it's possible save and restore the kv_state after the completion has been generated as well.

[MillionthOdin16]

I don't understand why we don't just use interactive mode. Almost all the users coming from llama.CPP Are used to an interface where they send a message and get a quick response because there is no clearing of state between messages from the user, meaning there's also no need to reload the state. As I understood it, the KV cache was a way to store the prompt state, because it is used multiple times during the course of a conversation and helps improve responsiveness during long conversations. Given the way that it's used in the base llama.CPP executable, and the fact that in the current implementation of interactive mode, storing of the entire conversation state won't improve performance (it would only allow continuation of previous conversations during a different session), I don't know that this is something that they're going to add in the immediate future.

For me, being able to get completions from the bot with full context of the ongoing conversation is my main use case. So there's pretty much no situation where I would want the current conversation context cleared or reset.

And I thought this was similar for the openAI implementation, where you send the current message, but don't need to send the full message history. Any type of recomputation or loading of model state decreases the performance and makes it slower than the base llama.CPP implementation imo.

After all, I think if people are using chat mode, from a user perspective, they want a continuous and performant chat. Even if that means running models and independent contexts simultaneously, which reduces scalability in the short term without the ability to load and save the states.

[abetlen]

@MillionthOdin16 are you talking about the OpenAI server or just using the Llama class? For the actual OpenAI API each request is entirely independent of all other requests (e.g. you always send the full history to the /chat/completions endpoint) so you do need to reset the model each time. This is why I'm looking into the KV state solution so we can just reload the state if we've seen e.g the first n-1 messages of an n message chat get sent over.

If you're just looking for interactive mode though I believe that's been implemented in this example if you just want to use it in a program and don't care about the API https://github.com/abetlen/llama-cpp-python/blob/main/examples/low_level_api/low_level_api_chat_cpp.py

[MillionthOdin16]

I'm talking about the openAI server. My point is from the user perspective, the most important factor of chat completions is speed of responses. Unfortunately, llama.CPP takes longer to process an initial prompt the longer it is. So for the chat completions endpoint, this creates an issue because the longer the conversation is, the longer it's going to take to get a response. The reason we have the issue is because the CPP implementation of llama is different than the normal GPU implementations relating to processing of the prompt before it generates a response.

So I'm saying that the most efficient solution in this instance might be to not clear the context and save that processing time for each subsequent completion by keeping the session going. It diverges from how openAI implements it, but it's the only option we have right now. And chat completions isn't usable without it because it's too slow.

[MillionthOdin16]

I'm basically advocating for a temporary hack to prevent the context from being cleared during chat completions so that we get a significant performance boost, until either we get a proper state saving capability, or the prompt processing time issue is resolved.

The issue is frustrating because we're so close to having an API that is performant and chat capable, but there's just a couple things holding it back, and I'm advocating for a temporary hack to allow good performance, until we can actually properly implement it.

[0xdevalias]

Unfortunately, llama.CPP takes longer to process an initial prompt the longer it is

@MillionthOdin16 Is that still meaningfully so since the recent performance regressions appear to have been fixed?

• Fix proposed: Performance Discrepancy: gpt4all Faster than Optimized llama.cpp ggml-org/llama.cpp#603 (comment)
  • Avoid heavy V transpose operation + improvements ggml-org/llama.cpp#775
• Fix seemingly confirmed: Performance Discrepancy: gpt4all Faster than Optimized llama.cpp ggml-org/llama.cpp#603 (comment)
• Potentially also resolved by the above: Alpaca model is running very slow in llama.cpp compared to alpaca.cpp ggml-org/llama.cpp#677 (comment)
• Potentially also resolved by the above: I'm pegging CPU (./examples/chat.sh works very slowly) on a 5800X3D / u22 linux, anything that can be done? ggml-org/llama.cpp#735 (comment)

[MillionthOdin16]

Unfortunately, llama.CPP takes longer to process an initial prompt the longer it is

@MillionthOdin16 Is that still meaningfully so since the recent performance regressions appear to have been fixed?

• Fix proposed: Performance Discrepancy: gpt4all Faster than Optimized llama.cpp ggerganov/llama.cpp#603 (comment)

  • Avoid heavy V transpose operation + improvements ggerganov/llama.cpp#775

• Fix seemingly confirmed: Performance Discrepancy: gpt4all Faster than Optimized llama.cpp ggerganov/llama.cpp#603 (comment)

• Potentially also resolved by the above: Alpaca model is running very slow in llama.cpp compared to alpaca.cpp ggerganov/llama.cpp#677 (comment)

• Potentially also resolved by the above: I'm pegging CPU (./examples/chat.sh works very slowly) on a 5800X3D / u22 linux, anything that can be done? ggerganov/llama.cpp#735 (comment)

So that's not what I mean in this case. I created issue 603 on llama.cpp and now that we have that performance boost, it would be awesome to get as much boost in the API over here as we can. I meant the issue/undetermined cause here: ggml-org/llama.cpp#719

I've seen people more familiar with LLMs mention some oddities about the initial processing in the past, but haven't seen a straightforward explanation. As I understand, it appears llama.cpp has some differences between how it processes the initial data before generating tokens, and it's much slower than the transformer's implementation (CPU vs GPU aside).

So I was just saying that if we get a performance boost from that avenue, as well as the ability to store a conversation's state, a proper implementation will be much faster than atm, and we wouldn't need this workaround. Hope that makes sense.

Right now there's so many small, different issues going on in the main repo that it's hard to keep track of haha.

[oobabooga]

+1 to this. Many people are requesting this feature here: oobabooga/text-generation-webui#866

It would be nice to have a use_cache=True flag (or something similar) in Llama.generate.

[abetlen]

@oobabooga I'm still working on the caching API but for now I've added a reset option for Llama.generate which defaults to False, if you want to continue from the end of a previous generation just call with reset=False.

[oobabooga]

@abetlen is it safe to use reset=False at all times, or will that cause incorrect generations if you completely replace the prompt with a new one?