Issue with LangChain Agent: Email Sending Tool Not Following Established Rules

Question

Problem Description:

I’m developing an agent using LangChain, langchain_openai, and a custom tool to send emails. The tool, named mail_tool, is designed to follow a strict set of rules and a specific input format for sending emails. However, I’m encountering issues where the agent sometimes doesn’t follow the rules, invents email addresses, or fails to send the correct content.

Tool Rules and Input Format:

The mail_tool must strictly follow these rules:

1. Tool Usage: It should only be used when the user requests to send information via email and provides their email address and the required content.

2. No Invented Emails: It must not invent email addresses; it should only use the one provided by the user.

3. Input Format: The input is a string that contains a dictionary because the model only returns strings:

{
    'mail': 'user_email',  # The user's email address.
    'content': 'content_to_send'  # The information to be sent.
}

Relevant Code:

Here’s the code I’ve implemented so far:

import asyncio
from langchain_openai import ChatOpenAI
from langchain.tools import Tool
from langgraph.checkpoint.memory import MemorySaver
from langchain_core.messages import HumanMessage, AIMessage
from langgraph.graph import StateGraph, START, END, MessagesState
from langchain.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain.agents import create_openai_functions_agent, AgentExecutor
import json

# Model and tools configuration
model = ChatOpenAI(model="gpt-4", openai_api_key="KEY_HERE")

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a friendly assistant named Axel, and you will answer questions. You only provide answers of a maximum of 25 words. You respond in Spanish"),
    MessagesPlaceholder(variable_name="chat_history"),
    ("human", "{input}"),
    MessagesPlaceholder(variable_name="agent_scratchpad")
])

def send_mail_tool(data: str):
    try:
        parsed_data = json.loads(data)
        mail = parsed_data.get("mail")
        content = parsed_data.get("content")
        if mail and content:
            return "Email sent"
        else:
            return "Error: 'mail' or 'content' is missing"
    except Exception as e:
        return f"Error: {str(e)}"

mail_tool = Tool(
    name="send_mail_tool",
    func=send_mail_tool,
    description="""
        FOLLOW THESE RULES STRICTLY TO USE THIS TOOL:
        - Use this tool only when the user requests to send information via email and provides their email address and the required content.
        - Only use it for this purpose.
        - Do not invent email addresses; only use the one provided by the user.
        - Always, without exception, use the Input format.
        - this is the Input Format:
            The input must be a dictionary with the following keys:
            {
                'mail': 'user_email',  # The user's email address.
                'content': 'content_to_send'  # The information to be sent.
            }
    """
)

tools = [mail_tool]

agent = create_openai_functions_agent(llm=model, prompt=prompt, tools=tools)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=False)

workflow = StateGraph(state_schema=MessagesState)

async def call_agent(state: MessagesState):
    user_input = state["messages"][-1].content
    chat_history = state["messages"][:-1]
    response = await agent_executor.ainvoke({
        "input": user_input,
        "chat_history": chat_history
    })
    return {"messages": state["messages"] + [AIMessage(content=response["output"])]}

workflow.add_node("agent", call_agent)
workflow.add_edge(START, "agent")

memory = MemorySaver()
app = workflow.compile(checkpointer=memory)

config = {"configurable": {"thread_id": "user1234"}}

app.update_state(config, {"messages": [
    AIMessage(content="You are a friendly assistant named Axel, and you will answer questions. You only provide answers of a maximum of 25 words. You respond in Spanish")
]})

async def main():
    print("Assistant Online. Type 'exit' to quit.")
    while True:
        user_input = input("\nYou: ")
        if user_input.lower() == "exit":
            print("Goodbye!")
            break
        app.update_state(config, {"messages": [HumanMessage(content=user_input)]})
        state = app.get_state(config).values
        try:
            output = await app.ainvoke(state, config)
        except Exception as e:
            print("\nAssistant: Sorry, an error occurred while processing your request.")
            print(f"Details: {str(e)}")
            continue
        print("\nAssistant:")
        print(output["messages"][-1].content)

if __name__ == "__main__":
    asyncio.run(main())

Specific Issue: The agent sometimes doesn’t follow the rules established for the mail_tool. Specifically:

1. Invents Email Addresses: Sometimes, the agent invents email addresses instead of using the one provided by the user.

2. Fails to Send Content: Other times, the agent doesn’t send the content correctly or doesn’t adhere to the specified input format.

Question:

How can I fix this behavior to ensure the agent strictly follows the rules and input format for the mail_tool? Are there any alternative approaches or improvements I can implement to make the agent more consistent?

Additional Information:

1. I’m using gpt-4 as the language model.

2. The send_mail_tool function is a simulation and doesn’t send real emails, but it should follow the specified format and rules.

I appreciate any suggestions or help you can provide. Thank you!

Answer 1

sometimes i find that the behavior changes and operates better when you use a better model. Try gpt-4o instead if you have not already.