how to use a fine tuned model in from langchain_community.llms.Ollama

Question

How to use a fine-tuned model in Ollama langchain_community.llms.ollama.Ollama. The correct way of passing any model is passing in model : str variable.

But the selection is only limited to https://ollama.com/library.

How can I add a fine-tuned gemma model as a string parameter.

I followed this video Ollama - Loading Custom Models , where he is able to add Quantized version of LLM into mac client of Ollama.

My use case is to fine tune a gemma:2b model, and save it to S3, and use this model in a compute instance as an API. My question revolves around how to intake this model in Ollama instance

• reading model from a path in compute instance
• Does the model require any quantization on top of it

Answer 1

I find a way to load custom local model ChatOllma(from langchain_community.chat_models import ChatOllama)

1. You need to covert the fine-tuned model via llama.cpp, and write a Modelfile to use the coverted model. ref. to: https://github.com/ollama/ollama/blob/main/docs/import.md

start serve

ollama serve

3. Create the ollama model:

ollama create \<model name> -f \<Modelfile path> 

4. Load the custom model in the python codes just like:

    self.llm = ChatOllama(model=<model name>)
    prompt = ChatPromptTemplate.from_template("Tell me a short joke about {topic}")
    chain = prompt | self.llm | StrOutputParser()
    print(chain.invoke({"topic": "Space travel"}))
   

It works for me. I hope it is useful to you ~!

Answer 2

You can upload your custom model to HuggingFace, create a Modelfile (https://github.com/ollama/ollama/blob/main/docs/modelfile.md), then build the model with ollama

Here's a reference > https://github.com/ollama/ollama