Alhusain Alharthi
Alhusain Alharthi

Reputation: 73

Issue Loading GGUF Model with ctransformers: RuntimeError on from_pretrained()

I'm trying to load a GGUF model (Hermes-3-Llama-3.1-8B.Q4_K_M.gguf) using the ctransformers Python package. I'm using the following code:

from ctransformers import AutoModelForCausalLM

llm = AutoModelForCausalLM.from_pretrained("Hermes-3-Llama-3.1-8B.Q4_K_M.gguf", model_type="llama")

However, I'm encountering the following error:

RuntimeError: Failed to create LLM 'llama' from 'Hermes-3-Llama-3.1-8B.Q4_K_M.gguf'.

Note: I am using ctransformers version 0.2.27 (latest version). I have downloaded the Hermes-3-Llama-3.1-8B.Q4_K_M.gguf model and placed it in the same folder of the app.py.

Upvotes: 1

Views: 186

Answers (0)

Related Questions