CODEWITHSUNDEEP
pythonlangchainpy-langchain
redmage123
redmage123

Reputation: 533

How to use the new gpt-3.5-16k model with langchain?

I have written an application in langchain that passes a number of chains to a Sequential Chain to run. The problem I'm having is that the prompts are so large that they are exceeding the 4K token limit size. I saw that OpenAI has released a new 16K token window sized model for ChatGPT, but I can't seem to access it from the API. When I try, I get the following error:

openai.error.InvalidRequestError: This is a chat model and not supported in the v1/completions endpoint. Did you mean to use v1/chat/completions?

Here is how I'm attempting to instantiate the model:

  self.llm = OpenAI(model='gpt-3.5-turbo-16k',temperature = self.config.llm.temperature,
                          openai_api_key = self.config.llm.openai_api_key,         
                          max_tokens=self.config.llm.max_tokens                 
                         )

Anybody know how I can fix this?

Upvotes: 10

Views: 15675

Answers (2)

Rijoanul Hasan Shanto
Rijoanul Hasan Shanto

Reputation: 944

Update 20th Auguest, 2023

OpenAIChat is deprecated. Use ChatOpenAI() model instead. Also one side note, the model name should pass through the model_name parameter.

from langchain.chat_models import ChatOpenAI

chat = ChatOpenAI(
    model_name='gpt-3.5-turbo-16k',
    temperature = self.config.llm.temperature,
    openai_api_key = self.config.llm.openai_api_key,         
    max_tokens=self.config.llm.max_tokens
)

Old Answer:

The gpt-3.5-turbo-16k is supposed to be used with the chat completion API endpoint. See the ref below:

ENDPOINT MODEL NAME
/v1/chat/completions gpt-4, gpt-4-0613, gpt-4-32k, gpt-4-32k-0613, gpt-3.5-turbo, gpt-3.5-turbo-0613, gpt-3.5-turbo-16k, gpt-3.5-turbo-16k-0613
/v1/completions text-davinci-003, text-davinci-002, text-curie-001, text-babbage-001, text-ada-001

So, instead of using the OpenAI() llm, which uses text completion API under the hood, try using OpenAIChat(). See the below example with ref to your sample code:

from langchain.llms import OpenAIChat

self.llm = OpenAIChat(
    model_name='gpt-3.5-turbo-16k',
    temperature = self.config.llm.temperature,
    openai_api_key = self.config.llm.openai_api_key,         
    max_tokens=self.config.llm.max_tokens                 
)

Upvotes: 7

Choppy
Choppy

Reputation: 402

from langchain.llms import OpenAI
llm = OpenAI(temperature=0.1, model_name="gpt-3.5-turbo-16k")

Works fine for me.

Upvotes: 2

Related Questions