(Discussion) Improve usability of llama-server

[ngxson]

While working on #13365 , I'm thinking about the use case where people can control llama-server completely via web UI, this including load/unload models and turning off the server.

The reason why I think about this idea is because I recently found myself going back to LM Studio quite often 😂 . llama.cpp server is good, but having to go back and forth between web <> CLI is not always a pleasant experience.

Basically I'm thinking about 3 low-hanging fruits that could improve the situation:

Idea 1: allow loading / unloading model via API: in server.cpp, we can add a kinda "super" main() function that wraps around the current main(). The new main will spawn an "interim" HTTP server that expose the API to load a model. Ofc this functionality will be restricted to local deployment to avoid any security issues.

Idea 2: add a -d, --detach flag to make the CLI go "headless", so user can close the terminal and server can keep running on background. It should be trivial to do on Mac and Linux, but may requires some efforts on Windows. We can add an API to terminate the server process.

Idea 3: we can make a desktop shortcut that opens the web browser to llama.cpp localhost page. Basically this will make llama.cpp to become an "app" on the desktop, without spending too much efforts on our side. This is nice-to-have, but just noting here to see if anyone else have a better idea.

WDYT @ggerganov @slaren ?

Also tagging @cebtenzzre if you have any suggestions for the -d, --detach mode

[slaren]

I think that would be great, it would help bring llama.cpp to more people. It is also good to exercise this kind of full-application functionality in our own code, because then things like handling errors gracefully become mandatory, and I think there are many cases where llama.cpp still crashes when opening a model or choosing the wrong parameters. I suspect that other applications like LM Studio handle this by spawning the llama.cpp process separately and restarting it if it crashes, but we should do better than that.

[pwilkin]

I'm literally just writing a wrapper to swap llama.cpp configurations and make it emulate both LM Studio and Ollama (since IntelliJ Assistant only works with LM Studio and Copilot only works with Ollama :>

[JohannesGaessler]

One feature that I would have found useful for Elo HeLLM is if the server could more efficiently parallelize requests across multiple GPUs. What I ended up doing is to just spawn multiple server processes that each received an equal percentage of the prompts. It would be nice if I could instead spawn a single server process which then automatically balances the load across the GPUs. This is something that maybe makes more sense to implement at the ggml backend level though.

[JohannesGaessler]

More generally, loading/unloading models via an API would also be nice (but for my use case this presupposes that I can get by with only a single server process). Maybe something like --model-dir to determine the models that are available?

[kth8]

I think dynamically loading models via API is already accomplished by llama-swap.

[ngxson]

More generally, loading/unloading models via an API would also be nice (but for my use case this presupposes that I can get by with only a single server process). Maybe something like --model-dir to determine the models that are available?

The --model-dir can be useful if the model can be contained in single file, but in case of vision model it may not be very practical.

But anw, I already thought of this use case in #13202 ; basically now we will have a registry-based model manager like docker/ollama, so this means we can display a list of available models ready to be loaded.

And in far in the future, we could also allow user to search for online models via HF API (which can be done 100% on web UI), much like how it's currently done in LM Studio

[ggerganov]

A use case for llama.vscode is to be able to start multiple llama-server instances at the same time on different ports. For example, with the latest version we would need 3 separate servers:

• FIM (e.g. Qwen 2.5 Coder)
• Chat (e.g. Qwen 3)
• Embeddings (e.g. BERT)

It might be complicated to support this use case, so it's probably low priority. But it's something to think about too.

Anyway, great ideas (also #13385) - let's improve the usability!

[ggerganov]

Idea 2: add a -d, --detach flag to make the CLI go "headless"

This seems like should be completely solved by the operating system (e.g. systemd on Ubuntu, LaunchAgents on MacOS, Windows Services, etc.).

[ngxson]

This seems like should be completely solved by the operating system (e.g. systemd on Ubuntu, LaunchAgents on MacOS, Windows Services, etc.).

Hmm yeah this will requires a way to install the binary as a system service though (which won't be a low-hanging fruit.) I'm not entirely sure what is the best way to do this on windows and mac, so if someone more experienced about this, please feel free to leave suggestions

[rgerganov]

A use case for llama.vscode is to be able to start multiple llama-server instances at the same time on different ports.

I think the admin API for loading models which is proposed here should also run on dedicated network interface (localhost by default) and port. Having a clear separation of admin vs. user APIs will make the deployment easier in some cases. Later the admin API could be extended to start multiple instances of llama-server

[Dampfinchen]

I think dynamically loading models via API is already accomplished by llama-swap.

Yeah but making that accessible in the main project in a user friendly way would be a huge deal, especially for Windows users who do not have docker and the like installed.

[kth8]

I think dynamically loading models via API is already accomplished by llama-swap.

Yeah but making that accessible in the main project in a user friendly way would be a huge deal, especially for Windows users who do not have docker and the like installed.

You don't need Docker to use llama-swap, in fact the author provide precompiled Windows binary of it.

[ericcurtin]

This seems like should be completely solved by the operating system (e.g. systemd on Ubuntu, LaunchAgents on MacOS, Windows Services, etc.).

Hmm yeah this will requires a way to install the binary as a system service though (which won't be a low-hanging fruit.) I'm not entirely sure what is the best way to do this on windows and mac, so if someone more experienced about this, please feel free to leave suggestions

podman does this kinda thing, it uses the -d flag. Technically you don't really have to implement this, the user can always do things like nohup llama-server &.

What I will say is try to be somewhat like Ollama if we can, then lots of tools like Open WebUI will "just work"

[ericcurtin]

I think that would be great, it would help bring llama.cpp to more people. It is also good to exercise this kind of full-application functionality in our own code, because then things like handling errors gracefully become mandatory, and I think there are many cases where llama.cpp still crashes when opening a model or choosing the wrong parameters. I suspect that other applications like LM Studio handle this by spawning the llama.cpp process separately and restarting it if it crashes, but we should do better than that.

By the way ramalama in the latest release uses vanilla llama-server for everything now, we stopped using llama-run, so if you have the llama.cpp binaries installed, you can just do:

ramalama --nocontainer run gemma3
ramalama --nocontainer serve gemma3
etc.

for basic testing.