Reputation: 73
I'm trying to load a GGUF model (Hermes-3-Llama-3.1-8B.Q4_K_M.gguf
) using the ctransformers
Python package. I'm using the following code:
from ctransformers import AutoModelForCausalLM
llm = AutoModelForCausalLM.from_pretrained("Hermes-3-Llama-3.1-8B.Q4_K_M.gguf", model_type="llama")
However, I'm encountering the following error:
RuntimeError: Failed to create LLM 'llama' from 'Hermes-3-Llama-3.1-8B.Q4_K_M.gguf'.
Note: I am using ctransformers version 0.2.27 (latest version). I have downloaded the Hermes-3-Llama-3.1-8B.Q4_K_M.gguf model and placed it in the same folder of the app.py.
Upvotes: 1
Views: 186