Langchain - ConversationalRetrievalChain with memory and customized prompt

Question

I'm trying to create a ConversationalRetrievalChain to answer based on a specific context provided by a pdf file. I can get good answers. The issue is that the memory is not working.

The code:

template2 = """
    Your name is Bot.
    You are a chatbot specialized in human resources. 
    Use the following context (delimited by <ctx></ctx>) to answer the questions.
    If there is any history of previous conversations, use it to answer (delimited by <hs></hs>)
    If you don't know the answer just answer that you don't know. 
    ------
    <ctx>
    {context}
    </ctx>
    ------
    <hs>
    {chat_history}
    </hs>
    ------
    Question:
    {question} 
    """
    
    
prompt2 = PromptTemplate(
                template=template2, 
                input_variables=["context", "chat_history", "question"])


def querying_V1(query, chat_history,prompt):
     memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True, output_key='answer')

    conv_chain = ConversationalRetrievalChain.from_llm(
          llm=llm,
          chain_type="stuff",
          retriever=vectorstore.as_retriever(search_kwargs={"k": 2}), 
          memory=memory,
          # condense_question_prompt=prompt,
          combine_docs_chain_kwargs={"prompt": prompt},
          output_key='answer',
          # return_source_documents=True,
          get_chat_history=lambda h : h,
          verbose = False
    )

    result = conv_chain({"question": query, "chat_history": chat_history})
    return result["answer"].strip()

while True:
    query = input("Prompt: ")
    if query == "q":
         sys.exit()
    result = querying_V1(query,chat_history,prompt2)
    print("\n" + result)
    chat_history.append((query, result))

This works to answer questions based on the context provided in the vectorstore. But if I ask for instance "what was the last question that I made?", the result is "I don't know".

I've read here Why doesn't langchain ConversationalRetrievalChain remember the chat history, even though I added it to the chat_history parameter? that if the ConversationalRetrievalChain object is being created in every iteration of the while loop, the new memory will overwrite the previous one.

If I define the memory and the conv_chain outside the function and call the conv_chain as input:

def querying_V2(query : str, conv_chain: object, chat_history):
    result = conv_chain({"question": query, "chat_history": chat_history})
    return result["answer"].strip()

The memory works but it seems to forget the context passed on the prompt...

I've already tried to change the result for result = conv_chain({"question": query) and some other suggestions based on similar issues. But until now, I was not able to have the ConversationalRetrievalChain answering based on the context provided in the prompt template and also based on the chat_history.

Answer 1

I know I am late, but looking at your code I can tell what is the problem, may be this will help others. In your code, the memory initialization is happening as local variable to that particular function. So, every time you are making a call to 'querying_V1' function a new memory object (ConversationBufferMemory) getting initialized. If you are using Streamlit api, below change will help you fix the issue:

def querying_V1(query, chat_history,prompt):
     if "memory" not in st.session_state:
          st.session_state.memory = ConversationBufferMemory(memory_key="chat_history", return_messages=True, output_key='answer')

     conv_chain = ConversationalRetrievalChain.from_llm(
          llm=llm,
          chain_type="stuff",
          retriever=vectorstore.as_retriever(search_kwargs={"k": 2}), 
          memory=st.session_state.memory,
          # condense_question_prompt=prompt,
          combine_docs_chain_kwargs={"prompt": prompt},
          output_key='answer',
          # return_source_documents=True,
          get_chat_history=lambda h : h,
          verbose = False
    )