How to use ConversationSummaryMemory with RunnableWithMessageHistory in LangChain v0.2?

Question

With LLMChain deprecated in LangChain v0.2, I am struggling to get ConversationSummaryMemory working again.

My chatbot is using RunnableWithMessageHistory with FileChatMessageHistory like this:

prompt = ChatPromptTemplate.from_messages([
    MessagesPlaceholder(variable_name="messages"),
    HumanMessagePromptTemplate.from_template("{content}"),
])

chain = prompt | chat;

def get_session_history(session_id: str) -> BaseChatMessageHistory:
    return FileChatMessageHistory(f"messages_{session_id}.json");

with_message_history = RunnableWithMessageHistory(
    chain, 
    get_session_history=get_session_history,
    input_messages_key="content",
    history_messages_key="messages",
);

while True:
    content = input(">> ");
    result = with_message_history.invoke(
        input={
            "content": content,
        },
        config={
            "configurable": {"session_id": "abc123"}
        }
    );
    print(result.content);

Instead of remembering all messages, I would like them summarized. With LLMChain I was able to use ConversationSummaryMemory. Now in v0.2 I cannot use it with RunnableWithMessageHistory, because ConversationSummaryMemory is not a subclass of BaseChatMessageHistory, which I could return from get_session_history().

What is the recommended best practice to remember summarized history with RunnableWithMessageHistory in LangChain v0.2?

Answer 1

Yeh, this was my best attempt at using ConversationSummaryMemory with RAG. It's got some of my library references so won't just run out of the box but will hopefully get you to where you want. Found it a bit messy as one of the other commenters mentioned. On a side note I've found with RAG the Information Retrieval process i.e. the query against the index is the weakest link. I think LLMs more than capable of summary info to give you answer but getting the right info in the first place is the tricky bit I found. Of course I'm just using naive RAG, I suspect people smarter than me using more sophisticated approaches.

from operator import itemgetter    
import openai
from langchain.chains.conversation.base import ConversationChain

from langchain.chains.retrieval_qa.base import RetrievalQA
from langchain.globals import set_debug, set_verbose
from langchain.memory import ConversationSummaryMemory
from langchain_community.vectorstores.faiss import FAISS
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.prompts.prompt import PromptTemplate
from langchain_core.runnables import (
    RunnableLambda,
    RunnablePassthrough,
)
from langchain_openai import OpenAIEmbeddings, ChatOpenAI

import OpenAIHelper
from llm_library import PrintCurrentPromptToConsoleCustomHandler

set_debug(True)
set_verbose(True)

# get your key here howwever you normally do it
# openai.api_key = OpenAIHelper.ApiKeySingleton.get_api_key()
embedding = OpenAIEmbeddings(model="text-embedding-3-small")
model = ChatOpenAI(model="gpt-3.5-turbo-0125", temperature=0.0)
store = {}


def pdf_retriever(query):
    global model
    assert isinstance(query, str)
    path = "C:/Users/leejo/Desktop/BT PDS/PDS + Adviser guide"
    retriever = FAISS.load_local(path, embedding,
                                 allow_dangerous_deserialization=True)
    qa = RetrievalQA.from_chain_type(llm=model, chain_type="stuff", retriever=retriever.as_retriever())
    results = qa.invoke(query)
    return results["result"]


pdf_retriever = RunnableLambda(pdf_retriever)

# The {input} field below is essentially used to feed into the pdf retriever
template = """Conversation history : {history}

Human: {input}
AI Assistant:"""

# Test prompt
'''
Conversation history : {history}

Human: {input}
AI Assistant:
'''

custom_conversation_prompt = PromptTemplate(input_variables=["history", "input"], template=template)

conversation = ConversationChain(
    prompt=custom_conversation_prompt,
    llm=model,
    verbose=True,
    memory=ConversationSummaryMemory(ai_prefix="AI Assistant", llm=model, input_key="input", memory_key="history")
)

prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You're an assistant who's good at insurance. Here is some context : {context}. "
            "Please answer any question the human asks you.",
        ),
        ("human", "{input}"),
    ]
)
context = itemgetter("input") | pdf_retriever
first_step = RunnablePassthrough.assign(context=context)

chain = conversation


def information_cli():
    while True:
        # Prompt the user to enter information
        user_input = input("Yes? (type 'stop' to exit): ")

        # Check if the user wants to stop the program
        if user_input.lower() == 'stop':
            print("Exiting the program.")
            break

        answer = chain.invoke(
            {"input": user_input},
            config={"callbacks": [PrintCurrentPromptToConsoleCustomHandler()]}
        )

        # lots of prints to help me understand how it all works :)
        print("*************************************************")
        print(f"ConversationSummary template : {conversation.memory.prompt.template}")
        print("--------------------------------")
        print(f"Current Summary: {conversation.memory.buffer}")
        print("--------------------------------")
        print(f"Conversation Memory Prompt: {conversation.memory.prompt}")
        print("--------------------------------")
        print(f"The answer is : {answer}")
        print(f"The answer is : {answer['response']}")
        print("--------------------------------")


information_cli()