ChatGroq with pythonrepl agent tool returning random answers

Question

I have a document chatbot application written in Python that uses Langchain and OpenAIEmbedding to upload & chunk documents (PDF, Word, etc) into SupabaseVectorStore database table called 'documents', which has the columns id, content, metadata and embedding. When querying it, it employs PythonREPL agent tools to finetune the answers. For instance, if you as "How many times does the word 'project' occur in the context", it uses the custom WordCount tool to calculate it, otherwise it uses pythonrepl.run(). The issue is that if you query something like "generate a bulleted list of the top 5 key points in the context", the response is not always consistent and often includes JSON response from the tools used, sometimes not even returning anything, or an error. Sometimes when I ask for a word count, it might respond "We will use the WordCount tool for this query...", instead of just returning the answer.

The function is included below

def query_document():
    if request.method == 'OPTIONS':
        return handle_options_request()
    
    data = request.json
    query = data.get('query')
    user_id = session.get('user_id', 'Not set')
    filenames = data.get('filenames', [])

    if not query:
        return jsonify({"error": "No query provided"}), 400
    
    vector_store = SupabaseVectorStore(
        client=supabase,
        embedding=embeddings,
        table_name="documents",
        query_name="match_documents",
        chunk_size=500
    )
    
    similarity_filter = {
        "user_id": user_id,
    }
 
    results = vector_store.similarity_search(
        query,
        filter=similarity_filter,
        k=None
    )
    
    if not results:
        return jsonify({"error": "No matching documents found in similarity search"}), 404
        
    context = "\n".join([doc.page_content for doc in results])

    tools = [
        Tool(
            name="PythonREPL",
            func=PythonREPLTool().run,
            description="A Python REPL. Use this to execute python commands. Input should be a valid python command. If you want to see the output of a value, you should print it out with `print(...)`. Use this tool to perform calculations or execute Python code."
        ),
        Tool(
            name="WordCount",
            func=lambda x: word_count(context, x),
            description="Counts the occurrences of a word in the context. Input should be a single word."
        )
    ]

    llm = ChatGroq(
        temperature=0.1,
        model_name="llama3-8b-8192",
        streaming=True,
        callbacks=[StreamingStdOutCallbackHandler()]
    )
    conversational_memory = ConversationBufferWindowMemory(
        memory_key='chat_history',
        k=5,
        return_messages=True
    )
    agent = initialize_agent(tools, llm, agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION, verbose=True, handle_parsing_errors=True, max_iterations=10,
                             memory=conversational_memory)

    system_prompt = f"""You are an AI assistant for a document chatbot. Your primary task is to answer questions based on the provided context from uploaded documents. Follow these guidelines:

        1. Use only the information in the context to answer questions.
        2. If the answer isn't in the context, say "I don't have enough information to answer that question."
        3. Use the WordCount tool only for specific word frequency questions.
        4. Use the PythonREPL tool only for complex calculations or data manipulation that you cannot do mentally.
        5. For tasks like summarization, listing key points, or general comprehension questions, use your natural language processing abilities without relying on tools.
        6. Keep responses concise, clear, and directly related to the query.
        7. Format your responses using proper markdown syntax:
        - Use # for main headings, ## for subheadings, etc.
        - Use * or - for unordered lists
        - Use 1. 2. 3. for ordered lists
        - Use `backticks` for inline code and ``` for code blocks
        - Use **bold** and *italic* for emphasis
        8. If the response is a list, add a \n at the end of each list item.
        9. Ensure your markdown is properly formatted and will render correctly.

        Context:
        {context}
        Query:
        {query}
"""

    def generate():
        response = agent.run(system_prompt)
        for chunk in response.split():  # Simulating streaming for compatibility
            yield chunk + " "

    return Response(stream_with_context(generate()), content_type='text/plain')

What am I doing wrong? Can someone please point out how to fix this issue so that it only returns the final answer (always) and no additional tools stuff. Please help!