Closed
Description
cuBLAS with llama-cpp-python on Windows.
Well, it works on WSL for me as intended but no tricks of mine help me to make it work using llama.dll in Windows.
I try it daily for the last week changing one thing or another. Asked friend to try it on a different system but he found no success either.
At this point i sincerely wonder if anyone ever made this work.
I don't expect any help regarding this issue, but If anyone could confirm that it does work in Windows and is possible at all as a concept it would surely encourage me to continue this struggle.
(there is no issue to make original llama.cpp to work with cuBLAS anywhere, issue lies with a wrapper for me)
I'll post error just in case:
File E:\LLaMA\llama-cpp-python\llama_cpp\llama_cpp.py:54, in _load_shared_library(lib_base_name)
53 try:
---> 54 return ctypes.CDLL(str(_lib_path))
55 except Exception as e:
File ~\AppData\Local\Programs\Python\Python310\lib\ctypes\__init__.py:374, in CDLL.__init__(self, name, mode, handle, use_errno, use_last_error, winmode)
373 if handle is None:
--> 374 self._handle = _dlopen(self._name, mode)
375 else:
FileNotFoundError: Could not find module 'E:\LLaMA\llama-cpp-python\llama_cpp\llama.dll' (or one of its dependencies). Try using the full path with constructor syntax.
During handling of the above exception, another exception occurred:
RuntimeError Traceback (most recent call last)
Cell In[1], line 1
----> 1 from llama_cpp import Llama
2 llm = Llama(model_path="./models/7B/ggml-model.bin")
3 output = llm("Q: Name the planets in the solar system? A: ", max_tokens=32, stop=["Q:", "\n"], echo=True)
File E:\LLaMA\llama-cpp-python\llama_cpp\__init__.py:1
----> 1 from .llama_cpp import *
2 from .llama import *
File E:\LLaMA\llama-cpp-python\llama_cpp\llama_cpp.py:67
64 _lib_base_name = "llama"
66 # Load the library
---> 67 _lib = _load_shared_library(_lib_base_name)
69 # C types
70 llama_context_p = c_void_p
File E:\LLaMA\llama-cpp-python\llama_cpp\llama_cpp.py:56, in _load_shared_library(lib_base_name)
54 return ctypes.CDLL(str(_lib_path))
55 except Exception as e:
---> 56 raise RuntimeError(f"Failed to load shared library '{_lib_path}': {e}")
58 raise FileNotFoundError(
59 f"Shared library with base name '{lib_base_name}' not found"
60 )
RuntimeError: Failed to load shared library 'E:\LLaMA\llama-cpp-python\llama_cpp\llama.dll': Could not find module 'E:\LLaMA\llama-cpp-python\llama_cpp\llama.dll' (or one of its dependencies). Try using the full path with constructor syntax.
And here the Llama DLL that doesn't work:
llama.zip