-
Notifications
You must be signed in to change notification settings - Fork 11.9k
server : add some missing env variables #9116
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
This overall feature is very useful! |
@Nexesenex Currently we can't pass enum as environment variable, so for now I can't add The hacky solution is to duplicate the code from |
@ngxson I tried and reached the problem, hence my request. |
* server : add some missing env variables * add LLAMA_ARG_HOST to server dockerfile * also add LLAMA_ARG_CONT_BATCHING
* server : add some missing env variables * add LLAMA_ARG_HOST to server dockerfile * also add LLAMA_ARG_CONT_BATCHING
* server : add some missing env variables * add LLAMA_ARG_HOST to server dockerfile * also add LLAMA_ARG_CONT_BATCHING
* server : add some missing env variables * add LLAMA_ARG_HOST to server dockerfile * also add LLAMA_ARG_CONT_BATCHING
* server : add some missing env variables * add LLAMA_ARG_HOST to server dockerfile * also add LLAMA_ARG_CONT_BATCHING
Cont #9105
I forgot
LLAMA_ARG_HOST
andLLAMA_ARG_PORT
As a nice-to-have,
LLAMA_ARG_HF_REPO
andLLAMA_ARG_MODEL_URL
are also added. Although it's not used by HF inference endpoint, it will be useful if someone want to deploy llama.cpp to stateless/server-less platforms like heroku or google cloud run.