CODEWITHSUNDEEP
langchainpy-langchain
Davide Luise
Davide Luise

Reputation: 1

Using Vertex AI Model Garden with Langchain

I am trying to use Vertex AI Model Garden for inference with Langchain. I have successfully deployed Mistral to an endpoint in Google Cloud and want to get inference with the class VertexAIModelGarden which is already implemented. However, it seems the output returned by the endpoint is not correctly parsed as it only contains a single character (see image).

I found a pull request on the langchain github repo that mentions this issue and there is a link to another repo where someone has implemented a temporary fix ( https://github.com/shikanime/langchain-vertexai-extended ). I tried using this code and it seems to work but I was wondering if anyone else had this issue and if this "fix" will eventually be included in Langchain. Furthermore, the stream method is not yet supported in this fix so we can just use the invoke.

enter image description here

`def build_chain(self, temperature=0.1, max_tokens=100000): PROMPT = PromptTemplate.from_template(self.prompt)

 llm = VertexAIModelGardenVllm(project=self.projectid, 
 endpoint_id=self.endpid, location=self.region)
 self.chain = (
 doc": itemgetter("resume"), "question": itemgetter("job_description")}
 | PROMPT 
 | llm
 | StrOutputParser()
 `

Upvotes: 0

Views: 534

Answers (1)

Monobina Saha
Monobina Saha

Reputation: 1

Are you trying to create a retriever and generator? I think you are trying to feed the resume in your RAG and you are asking to get job_description? Please clarify your question.

Upvotes: 0

Related Questions