What is the difference between OpenAI and ChatOpenAI in LangChain?

Question

I read the LangChain Quickstart.

There is a demo inside:

from langchain.llms import OpenAI
from langchain.chat_models import ChatOpenAI

llm = OpenAI()
chat_model = ChatOpenAI()

llm.predict("hi!")
>>> "Hi"

chat_model.predict("hi!")
>>> "Hi"

I searched the rest of the document and also online, but didn't find any info for the difference between OpenAI and ChatOpenAI.

Based on from langchain.llms import OpenAI, OpenAI is a large language model (LLM) which is also chat related.

So is OpenAI more general-purpose, while ChatOpenAI more chat focused?

What is the difference between OpenAI class and ChatOpenAI class in LangChain? Could someone clarify?

Answer 1

TL;DR: Both objects are used to make calls to openai.OpenAI API endpoints. However, langchain_openai.OpenAI makes calls to a deprecated completions endpoint that is for older models such as DaVinci while langchain_openai.ChatOpenAI makes calls to the newer chat completions endpoint that is for newer models such as "gpt-4" or "gpt-4o".

---

Hongbo Miao's answer covers difference at a high level pretty well. This answer aims to show the difference on a more practical level as of langchain 0.1.14, langchain-core 0.1.37, langchain-openai 0.1.1 and openai 1.14.3.

What is the difference between the two when a call to invoke() is made?

With OpenAI, the input and output are strings, while with ChatOpenAI, the input is a sequence of messages and the output is a message. They use different API endpoints and the endpoint of OpenAI has received its final update in July 2023.

When using the LLM model (OpenAI), the following code:

from langchain_openai import OpenAI
llm = OpenAI()
question = "What is after Monday?"
llm.invoke(question)                # '\n\nAfter Monday comes Tuesday.'

goes through the following steps: question is converted into a StringPromptValue, which in turn is converted into a list of prompts similar to [[What is after Monday?]] and this is sent as a v1/completions post request to OpenAI API (<Request('POST', 'https://api.openai.com/v1/completions')), which creates a Completion object. A typical Completion object would look like the following:

Completion(id='cmpl-99csdzxcfghjklpoiuytrcvbnmkjhg', choices=CompletionChoice(
finish_reason='stop', index=0, logprobs=None, text='\n\nAfter Monday comes Tue
sday.')], created=1712078760, model='gpt-3.5-turbo-instruct', object='text_com
pletion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=6, 
prompt_tokens=5, total_tokens=11))

From the above object, the text response is parsed and cast into a string and returned.

On the other hand, when using the chat model (ChatOpenAI):

from langchain_openai import ChatOpenAI
llm = ChatOpenAI()
question = "What is after Monday?"
llm.invoke(question)                # AIMessage(content='Tuesday')

the call goes through the following steps: question is converted into a StringPromptValue, which in turn is converted into a list of dictionaries where roles are clearly defined, similar to [{'content': 'What is after Monday?', 'role': 'user'}] and this is sent as a v1/chat/completions post request to OpenAI API (<Request('POST', 'https://api.openai.com/v1/chat/completions')>), which creates a ChatCompletion object. A typical ChatCompletion object will look like the following:

ChatCompletion(id='chatcmpl-99csdzxcfghjklpoiuytrcvbnmkjhg', choices=[Choice(f
inish_reason='stop', index=0, logprobs=None, message=ChatCompletionMessage(con
tent='Tuesday', role='assistant', function_call=None, tool_calls=None))], crea
ted=1712079023, model='gpt-3.5-turbo-0125', object='chat.completion', system_f
ingerprint='fp_b28b39ffa8', usage=CompletionUsage(completion_tokens=1, prompt_
tokens=12, total_tokens=13))

From the above object, the message response is parsed and cast into an AIMessage object and returned.

In summary: With OpenAI, prompts are created out of a question and a Completion object is created via POST request to an OpenAI API endpoint while with ChatOpenAI, messages are created out of a question and a ChatCompletion object is created. As the above examples show, these make a request to different API endpoints of OpenAI. However, as the linked docs show, the Completions API endpoint has already received its final update on July 2023, so probably moving forward, its better to use ChatOpenAI.

Why does it matter?

The above examples show that the outputs are essentially the same for simple examples. However, since the API endpoints are different, there are some differences when you want to create a bit more complex models. For example, if you want to create agents, then depending on the agent type, you’ll need to use one or the other because the way to pass prompt, chat history etc. is configured differently for each agent.

For example, if you want to create an OpenAI tools agent, then you can do so only using ChatOpenAI() because its API endpoint accepts a tools parameter, while the API endpoint of OpenAI() doesn't.

The following is a minimal example where an OpenAI tools agent is created that uses a single tool that multiplies two numbers. With ChatOpenAI, not only can we create the agent and its tool, we can also allow the model to use chat history (in the example below the question itself is "tell me" which without chat history is senseless).

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.agents import create_openai_tools_agent, AgentExecutor
from langchain.tools import tool
from langchain_core.messages import HumanMessage, AIMessage

@tool
def multiply(a, b):
    "Multiply to numbers. For any questions about multiplying two numbers, you must use this tool!"
    return a * b

llm = ChatOpenAI()
tools = [multiply]

prompt = ChatPromptTemplate.from_messages([
  ("system", "You are a helpful assistant"),
  MessagesPlaceholder(variable_name="chat_history", optional=True),
  ("human", "{input}"),
  MessagesPlaceholder(variable_name="agent_scratchpad")
])

chat_history = [HumanMessage(content="Do you know what is 5 times 6?"), AIMessage(content="Yes!")]
agent = create_openai_tools_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools)
response = agent_executor.invoke({"input": "tell me", "chat_history": chat_history})    # the question itself is based on prior history
print(response["output"])        # 5 times 6 is 30.

Trying to do the same using OpenAI() will raise a TypeError: Completions.create() got an unexpected keyword argument 'tools' error.

On the other hand, if you want to create a ReAct agent, then you need to pass the OpenAI instance.

Answer 2

TL;DR

Based on my research,

• OpenAI class includes more generic machine learning task attributes such as frequency_penalty, presence_penalty, logit_bias, allowed_special, disallowed_special, best_of.

• ChatOpenAI class provides more chat-related methods, such as completion_with_retry, get_num_tokens_from_messages to make it more user-friendly when build chatbot related applications.

---

Class Inheritance

Upon reviewing the source code, here's what I've discovered.

Listed below are the class inheritances for both the OpenAI and ChatOpenAI classes, along with their respective class attributes and methods.

OpenAI

OpenAI ← BaseOpenAI ← BaseLLM ← BaseLanguageModel

ChatOpenAI

ChatOpenAI ← BaseChatModel ← BaseLanguageModel

Comparison

Let's begin our comparison, moving from the fourth column to the first column.

Fourth Column

Both classes ultimately inherit from the base class BaseLanguageModel.

Third Column

BaseLLM and BaseChatModel are very similar with slightly difference:

• For OpenAI's BaseLLM, it includes additional methods:

  • batch(self, inputs, config=None, max_concurrency=None, **kwargs)
  • abatch (self, inputs, config=None, max_concurrency=None,**kwargs)

• For ChatOpenAI's BaseChatModel, it includes an extra method:

  • _combine_llm_outputs(self, llm_outputs)

Second Column

The second column contains the BaseOpenAI class, which primarily exists due to the presence of higher-level classes OpenAI and AzureOpenAI. However, they all share the same class attributes and methods.

First Column

At the top-level class (first column):

• OpenAI class includes more generic machine learning task attributes such as frequency_penalty, presence_penalty, logit_bias, allowed_special, disallowed_special, best_of.

• ChatOpenAI class provides more chat-related methods, such as completion_with_retry, get_num_tokens_from_messages to make it more user-friendly when build chatbot related applications.