CODEWITHSUNDEEP
google-cloud-platformgoogle-cloud-vertex-aimistral-7b
Deckard
Deckard

Reputation: 307

How to use Vertex AI Model Garden models in Workbench?

I've managed to deploy Mistral-7B-v0.1 to a Vertex AI endpoint. For this I've used its model id in the Model Garden ("mistralai/Mistral-7B-v0.1"). But when I use this id for other things, it doesn't work, for example:

from vertexai.language_models import TextGenerationModel

embeddings_model = TextEmbeddingModel.from_pretrained("mistralai/Mistral-7B-v0.1")
text_model = TextGenerationModel.from_pretrained("mistralai/Mistral-7B-v0.1")

Causes an error with the message "Invalid PublisherModel resource name". What id or package should I use to be able to get things such as a TextEmbeddingModel from a model in the Model Garden?

I'm running this from a Vertex AI Workbench instance, so I would expect it to be up-to-date with everything in GCP.

Documentation and examples are very sparse -- most non-GCP sources don't use the Model Garden (yet?) and all the documentation and examples that Google provides itself are all focused on their own models (bison, gecko, etc.)

Upvotes: 0

Views: 678

Answers (1)

Othmane El Omri
Othmane El Omri

Reputation: 91

It seems to me that it's not supported directly. You will have to deploy it then use the endpoint to generate predictions like described in this notebook.

instances = [
    {
        "prompt": "My favourite condiment is",
        "n": 1,
        "max_tokens": 200,
        "temperature": 1.0,
        "top_p": 1.0,
        "top_k": 10,
    },
]
response = endpoint.predict(instances=instances)

for prediction in response.predictions:
    print(prediction)

You can also use the HuggingFace transformers library if you want to use it locally. An example is provided in the same notebook.

Upvotes: 1

Related Questions