How to use Google Vertex AI fine tuned model via Node.js

Question

I fine-tuned a model on Google Vertex AI. Before that, I was using regular models with this code(it works):

public static async SendMessage(prompt) {
    const vertexAI = new VertexAI({project: GOOGLE_PROJECT_ID, location: 'us-central1', googleAuthOptions: {keyFile: KEY_FILE_PATH}});

    const generativeModel = vertexAI.getGenerativeModel({
      model: 'gemini-1.0-pro',
    });

    try {
      const resp = await generativeModel.generateContent(prompt);
      const contentResponse = await resp.response;
      if(!contentResponse || !contentResponse.candidates || contentResponse.candidates.length == 0 || !contentResponse.candidates[0].content 
      || !contentResponse.candidates[0].content.parts || contentResponse.candidates[0].content.parts.length == 0) {
        throw Error("ERROR: NO RESPONSE RETURNED FROM GOOGLE GENAI")
      } else {
        return contentResponse.candidates[0].content.parts[0].text;
      }
    } catch(e) {
      console.error(e)
      return "Let's talk about this later."
    }
  }

Now, I'm trying to access my fine-tuned model with this code:

public static async SendMessage(prompt) {
    const ENDPOINT_URL = `https://us-central1-aiplatform.googleapis.com/v1/projects/${GOOGLE_PROJECT_ID}/locations/us-central1/endpoints/MY_ENDPOINT:predict`;

    const vertexAI = new VertexAI({project: GOOGLE_PROJECT_ID, location: 'us-central1', apiEndpoint: ENDPOINT_URL, googleAuthOptions: {keyFile: KEY_FILE_PATH}});

    const generativeModel = vertexAI.getGenerativeModel({
      model: 'FINE_TUNED_MODEL_NAME',
    });

    try {
      const resp = await generativeModel.generateContent(prompt);
      const contentResponse = await resp.response;
      if(!contentResponse || !contentResponse.candidates || contentResponse.candidates.length == 0 || !contentResponse.candidates[0].content 
      || !contentResponse.candidates[0].content.parts || contentResponse.candidates[0].content.parts.length == 0) {
        throw Error("ERROR: NO RESPONSE RETURNED FROM GOOGLE GENAI")
      } else {
        return contentResponse.candidates[0].content.parts[0].text;
      }
    } catch(e) {
      console.error(e)
      return "Let's talk about this later."
    }
  }

I created MY_ENDPOINT on Google Cloud console and deployed the model. But I'm not sure how to access it via a client(node.js in this case. It throws this error:

[2024-07-12T06:09:39.356Z] GoogleGenerativeAIError: [VertexAI.GoogleGenerativeAIError]: exception posting request to model
    at D:\Dev\Anima\Client\node_modules\@google-cloud\vertexai\build\src\functions\generate_content.js:49:15
    at process.processTicksAndRejections (d:\Dev\Anima\Client\lib\internal\process\task_queues.js:95:5)
    at async generateContent (D:\Dev\Anima\Client\node_modules\@google-cloud\vertexai\build\src\functions\generate_content.js:39:22)
    at async GoogleGenAI.SendMessage (d:\Dev\Anima\Client\Anima\GoogleGenAI.ts:20:20)
    at async GoogleGenAIController.Send (file:///D:/Dev/Anima/Client/jsbuild/Anima/GenAIController.js:30:24) {stackTrace: TypeError: fetch failed
    at node:internal…undici:12502:13
    at process.processTick…, name: 'GoogleGenerativeAIError', stack: 'GoogleGenerativeAIError: [VertexAI.GoogleGene…lient/jsbuild/Anima/GenAIController.js:30:24)', message: '[VertexAI.GoogleGenerativeAIError]: exception posting request to model'}

EDIT: I realized that vertexAI.getGenerativeModel method adds prefix "models.../publisher/google/..." to the model name before sending if you don't start your model name with "models/", so I need to pass full model path. But I'm not sure what it is. Also I'm not sure if ENPOINT is correct.

Answer 1

It's about how you pass the endpoint to VertexAI node.js API. We need to provide it to the model parameter like this: projects/PROJECT_ID/locations/us-central1/endpoints/ENDPOINT_ID

It's because how VertexAI node.js API deals with provided parameters. In post_request.js, this is how the final endpoint is generated: let vertexEndpoint = https://${vertexBaseEndpoint}/${apiVersion}/${resourcePath}:${resourceMethod}

Now, resourcePath is generated using the model parameter we pass to .getGenerativeModel method. We generally pass model names such as gemini-1.0-pro. In this case, it appends a prefix to it to create resourcePath which is the full path to Google's original model. When we want to use our fine-tuned models deployed to an endpoint in our Google Cloud project, we need to pass the endpoint(not the model) to the model parameter, as described above. In this case, Vertex node.js Api, checks to see if model parameter starts with 'projects/' and doesn't append any prefix.

Code:

const vertexAI = new VertexAI({project: GOOGLE_PROJECT_ID, location: 'us-central1', googleAuthOptions: {keyFile: KEY_FILE_PATH}});

    const generativeModel = vertexAI.getGenerativeModel({
      model: 'projects/PROJECT_ID/locations/us-central1/endpoints/ENDPOINT_ID',
    });

    try {
      const resp = await generativeModel.generateContent(prompt);
      const contentResponse = await resp.response;
      if(!contentResponse || !contentResponse.candidates || contentResponse.candidates.length == 0 || !contentResponse.candidates[0].content 
      || !contentResponse.candidates[0].content.parts || contentResponse.candidates[0].content.parts.length == 0) {
        throw Error("ERROR: NO RESPONSE RETURNED FROM GOOGLE GENAI")
      } else {
        return contentResponse.candidates[0].content.parts[0].text;
      }
    } catch(e) {
      console.error(e)
      throw Error(e)
    }