CODEWITHSUNDEEP
docker-composeartificial-intelligencealpaca
SealProgrammer
SealProgrammer

Reputation: 95

Dalai Alpaca install failing to run, invalid model file 'models/7B/ggml-model-q4_0.bin'

I started out trying to get Dalai Alpaca to work, as seen here, and installed it with Docker Compose by following the commands in the readme:

docker compose build
docker compose run dalai npx dalai alpaca install 7B
docker compose up -d

And it managed to download it just fine, and the website shows up. But when I try to prompt it, I get the error llama_model_load: invalid model file 'models/7B/ggml-model-q4_0.bin' (bad magic). Is there any way to fix this?

Full error in debug mode:

/root/dalai/alpaca/main --seed -1 --threads 6 --n_predict 200 --model models/7B/ggml-model-q4_0.bin --top_k 40 --top_p 0.9 --temp 0.8 --repeat_last_n 64 --repeat_penalty 1.3 -p "The expected response for a highly intelligent chatbot to "Hello!" is "" exit root@06ca5ae31973:~/dalai/alpaca# /root/dalai/alpaca/main --seed -1 --threads 6 --n_predict 200 --model models/7B/ggml-model-q4_0.bin --top_k 40 --top_p 0.9 --temp 0.8 --repeat_last_n 64 --repeat_penalty 1.3 -p "The expected response for a highly intelligent chatbot to "Hello!" is /root/dalai/alpaca/main --seed -1 --threads 6 --n_predict 200 --model models/7B/ggml-model-q4_0.bin --top_k 40 --top_p 0.9 --temp 0.8 --repeat_last_n 64 --repeat_penalty 1.3 -p "The expected response for a highly intelligent chatbot to "Hello""" is "" main: seed = 1684196106 llama_model_load: loading model from 'models/7B/ggml-model-q4_0.bin' - please wait ... llama_model_load: invalid model file 'models/7B/ggml-model-q4_0.bin' (bad magic) main: failed to load model from 'models/7B/ggml-model-q4_0.bin' root@06ca5ae31973:~/dalai/alpaca# exit exit

I have already tried reinstalling it. The file appears to be in the right place (dalai\models\alpaca\models), so I am unsure as to what is going wrong.

Upvotes: 5

Views: 4421

Answers (3)

Johannes Fahrenkrug
Johannes Fahrenkrug

Reputation: 44808

I got it to work by downloading the model from Hugging Face: https://huggingface.co/Sosaka/Alpaca-native-4bit-ggml/tree/main

I then copied it to ~/dalai/alpaca/models/7B and renamed the file to ggml-model-q4_0.bin.

I was then able to run dalai, or run a CLI test like this one:

~/dalai/alpaca/main --seed -1 --threads 4 --n_predict 200 --model models/7B/ggml-model-q4_0.bin --top_k 40 --top_p 0.9 --temp 0.8 --repeat_last_n 64 --repeat_penalty 1.3 -p "What color is the sky?"

Enjoy!

Upvotes: 3

Jack
Jack

Reputation: 1

I was not able to solve this problem too. I believe the cause is that the .bin model fails the magic verification which is checking the format of the expected model. I tried to change the model's first 4 bits to what it expects in magic verification error statement i.e "ggml" in ASCII. But that did not solve the problem. This indicates that the format of .bin file is wrong and probably dalai package failed at some step.

Upvotes: 0

John S
John S

Reputation: 91

I was having the same problem - I didn't exactly solve it but worked around it by using the instructions from one of the README.md files that was installed when I installed the nodejs/python based solution.

git clone https://github.com/antimatter15/alpaca.cpp

cd alpaca.cpp

make chat

The model file can be fetched with curl - three locations were given but this one was the fastest:

curl -o ggml-alpaca-7b-q4.bin -C - https://gateway.estuary.tech/gw/ipfs/QmQ1bf2BTnYxq73MFJWu1B7bQ2UD6qG7D7YDCxhTndVkPC

Then just run

./chat

It works for me but is quite slow and takes a lot of CPU!

Upvotes: 4

Related Questions