In Langchain, why ConversationalRetrievalChain not remembering the chat history and Entering new ConversationalRetrievalChain chain for each chat?

Question

I am trying to create an customer support system using langchain. I am using text documents as external knowledge provider via TextLoader

In order to remember the chat I using ConversationalRetrievalChain with list of chats

My problem is, each time when I execute conv_chain({"question": prompt, "chat_history": chat_history}),

it is creating a new ConversationalRetrievalChain that is, in the log, I get Entering new ConversationalRetrievalChain chain > message

And the chat_history array looks like, multiple nested arrays :

[[ "Hi I am Ragesh", "Hi Ragesh, How are your"] , ["What is my name?", "Sorry, As an AI....., " ]]

So it couldn't remember my previous chat.

Why this is happening ?

I am very new to AI field. Please help me.

My code:

import json

from colorama import Fore
from langchain.chains import ConversationalRetrievalChain
from langchain.chains.conversational_retrieval.base import BaseConversationalRetrievalChain
from langchain.chat_models import ChatOpenAI
from langchain.document_loaders import TextLoader
from langchain.embeddings import HuggingFaceInstructEmbeddings
from langchain.prompts import PromptTemplate
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.vectorstores import Chroma

chat_history = []
def generate_response(support_qa: BaseConversationalRetrievalChain, prompt):
    response = support_qa({"question": prompt, "chat_history": chat_history})
    chat_history.append((prompt, response["answer"]))
    print( json.dumps(chat_history))
    return response['answer']

if __name__ == '__main__':
    print("status: loading document")
    loader = TextLoader("./docs/doc.txt")
    pages = loader.load_and_split()

    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
        length_function=len,
    )

    docs = text_splitter.split_documents(pages)

    # Split documents into chunks
    gds_data_split = docs
    print(len(gds_data_split))

    # Define embedding model
    OPENAI_API_KEY = "sk-key"

    embeddings = HuggingFaceInstructEmbeddings(model_name="hkunlp/instructor-xl")
    support_data = gds_data_split
    support_store = Chroma.from_documents(
        support_data, embeddings, collection_name="support"
    )
    print("status: configure llm")
    llm = ChatOpenAI(
        model_name="gpt-3.5-turbo",
        temperature=0,
        openai_api_key=OPENAI_API_KEY,
        max_tokens=1024,
    )
    _template = """
    {chat_history}
    {question}"""
    CONDENSE_QUESTION_PROMPT = PromptTemplate.from_template(_template)

    support_template = """As a TechROVA marketing bot, your goal is to provide accurate and helpful information about TechROVA products,
    a software company selling softwares to clients.
    You should answer user inquiries based on the context provided and avoid making up answers.
    If you don't know the answer, go through the previous chats first and if not, then use your own training data.
    
    {context}

    Question: {question}"""

    SUPPORT_PROMPT = PromptTemplate(
        template=support_template, input_variables=["context", "question"]
    )
    print("status: confiure chain")
    support_qa = ConversationalRetrievalChain.from_llm(
        llm=llm,
        retriever=support_store.as_retriever(search_kwargs={"k": 3}),
        verbose=True,
        return_source_documents=True,
        qa_prompt=SUPPORT_PROMPT,
        condense_question_prompt=CONDENSE_QUESTION_PROMPT,
    )

    while True:
        user_input = input("-> ")
        response = generate_response(support_qa, user_input)
        print(Fore.RED + response)

Answer 1

support_qa = ConversationalRetrievalChain.from_llm(
        llm=llm,
        retriever=vectorstore.as_retriever(),
        # retriever=support_store.as_retriever(search_kwargs={"k": 3}),
        verbose=True,
        return_source_documents=True,
        combine_docs_chain_kwargs={"prompt": SUPPORT_PROMPT},
        # qa_prompt=SUPPORT_PROMPT,
        condense_question_prompt=CONDENSE_QUESTION_PROMPT,
    )

This can slove the question.

Answer 2

You need to update your Langchain version

Answer 3

you may add something here

def generate_response(support_qa: BaseConversationalRetrievalChain, prompt):
    response = support_qa({"question": prompt, "chat_history": chat_history})
    chat_history.append((prompt, response["answer"]))
    print( json.dumps(chat_history))
    return response['answer']

to this code below. for the time you need you history

def generate_response(support_qa: BaseConversationalRetrievalChain, prompt):
    chat_history = [(prompt, --**previous chat_history** --)]
    response = support_qa({"question": prompt, "chat_history": chat_history})
    chat_history.append((prompt, response["answer"]))
    print( json.dumps(chat_history))
    return response['answer']

this will ensure that support_qa chat_history is the history you were looking for. be carefull with token maximum issues. you may need to use map-reduce to summarize your history.