Closed
Description
Prerequisites
Please answer the following questions for yourself before submitting an issue.
- I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- I carefully followed the README.md.
- I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
- I reviewed the Discussions, and have a new bug or useful enhancement to share.
Expected Behavior
When freeing the model and verbose=False, no logging message should appear on stdout/stderr.
Current Behavior
ggml_metal_free: deallocating
is displayed
Environment and Context
Steps to Reproduce
In [1]: from llama_cpp import Llama
In [2]: class A:
...: def __init__(self):
...: self.model = Llama('/Users/pas/.cache/py-llm-core/models/mistral', verbose=False, n_ctx=4000, n_gpu_layers=100, n_threads=4)
...: def generate(self, prompt):
...: return self.model.create_completion(prompt)
...: def free_model(self):
...: del self.model
...:
In [3]: instance = A()
In [4]: instance.generate('Write a recipe to make a pizza')
Out[4]:
{'id': 'cmpl-76b2943e-1169-4062-bb6b-8e916279b329',
'object': 'text_completion',
'created': 1696768489,
'model': '/Users/pas/.cache/py-llm-core/models/mistral',
'choices': [{'text': ' crust that is perfect for a thin and crispy pizza.\n\nIngredients:\n- 3 cups all-purpose flour\n- 1 package active dry yeast\n- 2 tablespoons olive oil\n- 1 teaspoon salt\n- 1 cup warm water\n\nInstructions:\n\n1) In a large mixing bowl, combine the flour and yeast.\n\n2) Add the olive oil and salt to the mixture and stir until well combined.\n\n3) Pour in the warm water and use your hands to knead the dough until it forms a smooth ball.',
'index': 0,
'logprobs': None,
'finish_reason': 'length'}],
'usage': {'prompt_tokens': 8, 'completion_tokens': 128, 'total_tokens': 136}}
In [5]: instance.free_model()
ggml_metal_free: deallocating
Metadata
Metadata
Assignees
Labels
No labels