When I try to load the 'qwen' gguf file, it shows the error message: "error loading model: unknown model architecture: 'qwen'". Does this mean that the 'qwen' model is not supported?

[dgo2dance]

When I try to load the 'qwen' gguf file, it shows the error message: "error loading model: unknown model architecture: 'qwen'". Does this mean that the 'qwen' model is not supported?

[ouyyy18]

+1
error loading model: unknown model architecture: 'qwen'
llama_load_model_from_file: failed to load model

[qinxuye]

llama-cpp supported last week: ggml-org/llama.cpp#4281
Python binding needs to be updated.

[abetlen]

@dgo2dance sorry been away the past week but just had a chance to push this update, let me know if it works!

[awtestergit]

@dgo2dance sorry been away the past week but just had a chance to push this update, let me know if it works!

Not sure if this is the same issue, but when i tried to load the qwen model (either f16 or q8_0) using Ipython notebook in VSCode, the kernel dies.

llama_model = Llama(model_path = path, n_gpu_layers=-1, n_ctx=0)

There is not many error in logs:
14:24:22.976 [info] Dispose Kernel process 48352.
14:24:22.990 [info] End cell 1 execution @ undefined, started @ 1702448662120, elapsed time = -1702448662.12s

Command line works fine:
./main -m models/qwen14-chat-q8_0.gguf -p "Building a website can be done in 10 simple steps:\nStep 1:" -n 400 -e

Output:
Building a website can be done in 10 simple steps:
Step 1: Decide on the purpose of your website.
This is a very important step, and it will determine the content you put on your site. You need to think about why you want to build a website, what you want to achieve with it, and who your target audience is.
Step 2: Choose a domain name for your website.
Your domain name should be easy to remember and related to the purpose of your website. It’s also important to make sure that your domain name isn’t already taken by someone else.
Step 3: Find a web hosting company.
A web hosting company is a business that provides space on their servers for you to store your website files. You’ll also need to find out how much bandwidth they offer, as this will determine how many visitors your site can handle at one time.
Step 4: Design your website.
You don’t need to be a professional designer to create a good-looking website. There are plenty of tools and templates available that make it easy for anyone to design a website without any prior experience. Just make sure that you choose a layout that’s easy to read and navigate, and use colors that complement each other well.
Step 5: Create content for your website.
Your content is what will attract visitors to your site, so it needs to be high-quality and relevant. You can create text-based content, such as blog posts or articles, or you could include multimedia elements like videos or images. Just make sure that your content is well-written, informative, and engaging.
Step 6: Optimize your website for search engines.
Search engine optimization (SEO) is the process of making your website more visible in search results. There are a few things you can do to optimize your site for SEO, such as including relevant keywords in your content and building backlinks from other websites. You should also make sure that your site is mobile-friendly, since more and more people are accessing the internet on their smartphones.
Step 7: Test your website.

[awtestergit]

@abetlen

the issue seems to be from n_ctx parameter, if I set it to 0, then initializing qwen model crashed; initializing the original llama2 7b is ok but will throw ValueError at create_completion. seems to be n_batch?

if i did not set n_ctx then both load and complete fine.

File ~/anaconda3/envs/hfhub/lib/python3.11/site-packages/llama_cpp/llama.py:1057, in Llama.eval(self, tokens)
1055 assert self._batch.batch is not None
1056 self._ctx.kv_cache_seq_rm(-1, self.n_tokens, -1)
-> 1057 for i in range(0, len(tokens), self.n_batch):
1058 batch = tokens[i : min(len(tokens), i + self.n_batch)]
1059 n_past = self.n_tokens

ValueError: range() arg 3 must not be zero

[abetlen]

Closing this as I believe it was just related to the model not being supported in llama-cpp-python yet. I'll reopen if anyone is still experiencing this issue with the latest version.