How to persist LangChain conversation memory (save and load)?

Question

I'm creating a conversation like so:

llm = ChatOpenAI(temperature=0, openai_api_key=OPENAI_API_KEY, model_name=OPENAI_DEFAULT_MODEL)
conversation = ConversationChain(llm=llm, memory=ConversationBufferMemory())

But what I really want is to be able to save and load that ConversationBufferMemory() so that it's persistent between sessions. There doesn't seem to be any obvious tutorials for this but I noticed "Pydantic" so I tried to do this:

saved_dict = conversation.memory.chat_memory.dict()
cm = ChatMessageHistory(**saved_dict) # or cm = ChatMessageHistory.parse_obj(saved_dict)

But this fails:

ValidationError: 6 validation errors for ChatMessageHistory
messages -> 0
  Can't instantiate abstract class BaseMessage with abstract method type (type=type_error)

Thoughts? I'd love links to any sort of guide, repo, reference, etc.

Answer 1

"I solved the problem by saving conversation memories on the client side and adding them to the chat history on server side every time. It works great. The code on server side is much more clearner and flexible. You don't have to worry about session.

The code of persisting chat history on client side is straightforward, for exameple using LocalStorage of the browser."

How many previous chat history did you use to reformat the query? Because there is a context window limit for LLm right?

Answer 2

You can use FileChatMessageHistory https://api.python.langchain.com/en/latest/chat_message_histories/langchain_community.chat_message_histories.file.FileChatMessageHistory.html

from langchain_community.chat_message_histories import FileChatMessageHistory
...
# Load memory
memory = ConversationBufferMemory(return_messages=True, chat_memory=get_chat_history_local_file(file_id))

prompt = hub.pull("hwchase17/openai-tools-agent")
agent = create_openai_tools_agent(llm, tools, prompt=prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, memory=memory, verbose=True)

# Generate a response from the agent
response = agent_executor.invoke({"input":user_prompt}, )

def get_chat_history_local_file(conversation_id: str) -> FileChatMessageHistory:
    """Get a chat history from conversation id."""

    if not conversation_dir.exists():
        conversation_dir.mkdir(parents=True)
    file_path = conversation_dir / f"{conversation_id}.json"
    return FileChatMessageHistory(str(file_path))

There are also classes for other types of persistent storages: MongoDBChatMessageHistory, RedisChatMessageHistory, SQLChatMessageHistory, KafkaChatMessageHistory,DynamoDBChatMessageHistory,...

Answer 3

Adding to shum's answer, following is a git showing saving and passing memory for ConversationSummaryBuffer type. Pickle directly does not work on it as it contains multithreads.

Logic: Instead of pickling the whole memory object, we will simply pickle the memory.load_memory_variables({}) response. Later one can load the pickle object, extract the summary and conversation then pass it to newly instantiated memory object using following function:

for item in extract_mem['chat_history']:
    if isinstance(item, SystemMessage):
       blank_memory.moving_summary_buffer = item.content
    elif isinstance(item, HumanMessage):
       blank_memory.chat_memory.add_user_message(item.content)
    elif isinstance(item, AIMessage):
       blank_memory.chat_memory.add_ai_message(item.content)

https://github.com/Sidhant1201/langchain_ConversationSummaryBufferMemory

Answer 4

You should try using an SQL database to store your response and give a specific ID or Title to that conversation. Next time you want to query for the same conversation just give that ID or Title to SQL query retrieve the messages and load them to memory variable it could be ConversationBufferMemory().

I have done something similar in Django here is some code of it.

class Conversation(models.Model):
   title = models.CharField(max_length=100, unique=True)
   user = models.ForeignKey(User, on_delete=models.CASCADE)
   def __str__(self):
        return f"{self.user}:{self.title}"

class ChatMessage(models.Model):
    id = models.AutoField(primary_key=True)
    conversation = models.ForeignKey(Conversation, default=None, on_delete=models.CASCADE)
    user_response = models.TextField(null=True, default='')
    ai_response = models.TextField(null=True, default='')
    timestamp = models.DateTimeField(auto_now_add=True)

    def __str__(self):
        return f"{self.conversation}: {self.id}"

this help to create tables in sql db.

now we will see how to store messages and title in db:

def store_message(user_response, ai_response, conversation_id):
    ChatMessage.objects.create(
        user_response=user_response,
        ai_response=ai_response,
        conversation_id=conversation_id,
    )

def store_title(title, user):
    Conversation.objects.create(
        title=title,
        user=user
    )

Here is additional code to generate titles specific to your chats:

# API for title generator
API_URL = "https://api-inference.huggingface.co/models/czearing/article-title-generator"
headers = {"Authorization": f"Bearer YOUR_HF_TOKEN_FOR_ABOVE_API"}

def generate_title(payload):
    response = requests.post(API_URL, headers=headers, json=payload)
    return response.json()[0]['generated_text']

Here is the code to get conversations from the db and load it in the memory: `

def retrieve_conversation(title, user):
    # number of conversations
    num_recent_conversations = 4
    # Retrieve the most recent conversation history from the database
    conversation_obj = Conversation.objects.get(title=title, user=user)
    conversation_id = getattr(conversation_obj, 'id')
    
    # Retrieve recent conversation messages
    conversation_context = ChatMessage.objects.filter(
        conversation_id=conversation_id
    ).order_by('-timestamp')[:num_recent_conversations:-1]
    
    # Storing the retrived data from db to model memory 
    lst = []
    for msg in conversation_context:
        input_msg = getattr(msg, 'user_response')
        output_msg = getattr(msg, 'ai_response')
        lst.append({"input": input_msg, "output": output_msg})
    
    for x in lst:
        inputs = {"input": x["input"]}
        outputs = {"output": x["output"]}
        memory.save_context(inputs, outputs)
    
       
    retrieved_chat_history = ChatMessageHistory(
        messages=memory.chat_memory.messages
    )
    
    return retrieved_chat_history

Now we will use this ChatMessageHistory to give context to older conversations in the model

Following is my request function in my Django views that Sums up the code:

    elif request.method == 'POST':
        request_data = JSONParser().parse(request)
        prompt = request_data.get('prompt')
        user = request.user
        provided_title = request_data.get('title')
        if provided_title:
            # Create a ChatMessageHistory instance
            retrieved_chat_history = retrieve_conversation(
                provided_title, user)

        else:
            memory.clear()
            retrieved_chat_history = ChatMessageHistory(messages=[])

        reloaded_chain = ConversationChain(
            llm=llm,
            memory=ConversationBufferMemory(
                chat_memory=retrieved_chat_history),
            verbose=True
        )

        response = reloaded_chain.predict(input=prompt)

        if provided_title:
            title = provided_title
        else:
            # Generate a default title if not provided
            title = generate_title({
                "inputs": response
            })
            store_title(title, user)

        conversation_title = Conversation.objects.get(title=title, user=user)
        conversation_id = getattr(conversation_title, 'id')
        store_message(prompt, response, conversation_id)

        return JsonResponse({
            'ai_responce': response,
            'title':title
        }, status=201)

Hope this helps you

Answer 5

if you built a full-stack app and want to save user's chat, you can have different approaches:

1- you could create a chat buffer memory for each user and save it on the server. but as the name says, this lives on memory, if your server instance restarted, you would lose all the saved data. so this is not a real persistence.

2- the real solution is to save all the chat history in a database. when the user is logged in and navigates to its chat page, it can retrieve the saved history with the chat ID. since your app is chatting with open ai api, you already set up a chain and this chain needs the message history. so once you retrieve the chat history from the database, you should be populating chat chain memory with this chat history.

Answer 6

you try retrieve_from_db = json.loads(json.dumps(db_chain.memory.to_json()))

Answer 7

I just did something similar, hopefully this will be helpful. On a high level:

1. use ConversationBufferMemory as the memory to pass to the Chain initialization

llm = ChatOpenAI(temperature=0, model_name='gpt-3.5-turbo-0301')
original_chain = ConversationChain(
    llm=llm,
    verbose=True,
    memory=ConversationBufferMemory()
)
original_chain.run('what do you know about Python in less than 10 words')

2. extract messages from memory in the form of List[langchain.schema.HumanMessage|AIMessage] (not serializable)

extracted_messages = original_chain.memory.chat_memory.messages

3. transform the extracted message to serializable native Python objects

ingest_to_db = messages_to_dict(extracted_messages)

4. perform db operations to write to and read from database of your choice, I'll just use json.dumps and json.loads to illustrate

retrieve_from_db = json.loads(json.dumps(ingest_to_db))

5. transform the retrieved serialized object back to List[langchain.schema.HumanMessage|AIMessage]

retrieved_messages = messages_from_dict(retrieve_from_db)

6. construct ChatMessageHistory from the messages

retrieved_chat_history = ChatMessageHistory(messages=retrieved_messages)

7. construct ConversationBufferMemory from ChatMessageHistory

retrieved_memory = ConversationBufferMemory(chat_memory=retrieved_chat_history)

8. pass memory back to the newly initiated Chain

reloaded_chain = ConversationChain(
    llm=llm,
    verbose=True,
    memory=retrieved_memory
)

You can find the full code snippet at this GitHub Link

Answer 8

Rather than mess around too much with LangChain/Pydantic serialization issues, I decided to just use Pickle the whole thing and that worked fine:

pickled_str = pickle.dumps(conversation.memory)
conversation2 = ConversationChain(llm=llm, memory=pickle.loads(pickled_str)