CODEWITHSUNDEEP
pythonartificial-intelligencelangchainlarge-language-model
user24243499
user24243499

Reputation: 51

How to tune agent _executor for better understanding of the database

I have a database in which I have connected an agent too. However, I have noticed that it sometimes gets confused between whether or not it should return a column ID or persons first name when asked "which person sold the most....?" Is there a way to tune/adjust the create_sql_agent from langchain.agents at which I can tell the agent to not return column ID but return first and last name based on questions structured like that?

I think the question may be related to this post but I am unsure how to include that/and structure that properly: https://github.com/langchain-ai/langchain/discussions/9591

System Info langchain-openai==0.1.3 Python 3.11.7 Windows 11

Basic Model

from langchain_openai import ChatOpenAI
from langchain.agents.agent_toolkits import SQLDatabaseToolkit
from langchain.agents import create_sql_agent
from langchain.sql_database import SQLDatabase


llm = ChatOpenAI(model_name="gpt-3.5-turbo-1106", temperature=0, openai_api_key=os.environ.get('OPENAI_API_KEY'))

toolkit = SQLDatabaseToolkit(db=db, llm=llm)

agent_executor = create_sql_agent(
    llm=llm,
    toolkit=toolkit,
    verbose=False,
    agent_type="openai-tools")


print(agent_executor.invoke("What is my data about"))

Nothing, unsure how to progress as I can not find examples.

Upvotes: 0

Views: 626

Answers (1)

ThomaS
ThomaS

Reputation: 895

You can customize the prompts given to the LLM by passing a prompt argument to create_sql_agent function.

Looking at create_sql_agent source code at https://github.com/langchain-ai/langchain/blob/master/libs/community/langchain_community/agent_toolkits/sql/base.py

you can see if prompt is None it will create a prompt using:

messages = [
                SystemMessage(content=cast(str, prefix)),
                HumanMessagePromptTemplate.from_template("{input}"),
                AIMessage(content=suffix or SQL_FUNCTIONS_SUFFIX),
                MessagesPlaceholder(variable_name="agent_scratchpad"),
            ]
prompt = ChatPromptTemplate.from_messages(messages)

using prefix and suffix from there:

from langchain_community.agent_toolkits.sql.prompt import (
    SQL_FUNCTIONS_SUFFIX,
    SQL_PREFIX,
)

So you can do the same thing to create a prompt with more customized instructions.

And then pass it to the agent.

Here is an example with the prefix and suffix can explicitly be customized, to add a line about your issue.

from langchain_core.prompts.chat import (
    ChatPromptTemplate,
    HumanMessagePromptTemplate,
    SystemMessagePromptTemplate,
    AIMessagePromptTemplate,
    MessagesPlaceholder,
)

prefix = """
You are an agent designed to interact with a SQL database.
Given an input question, create a syntactically correct {dialect} query to run, then look at the results of the query and return the answer.
Unless the user specifies a specific number of examples they wish to obtain, always limit your query to at most {top_k} results.
You can order the results by a relevant column to return the most interesting examples in the database.
Never query for all the columns from a specific table, only ask for the relevant columns given the question.
You have access to tools for interacting with the database.
Only use the below tools. Only use the information returned by the below tools to construct your final answer.
You MUST double check your query before executing it. If you get an error while executing a query, rewrite the query and try again.

DO NOT make any DML statements (INSERT, UPDATE, DELETE, DROP etc.) to the database.

If the question does not seem related to the database, just return "I don't know" as the answer.
"""

suffix = """I should look at the tables in the database to see what I can query.  Then I should query the schema of the most relevant tables.
"""

messages = [
                SystemMessagePromptTemplate.from_template(prefix),
                HumanMessagePromptTemplate.from_template("{input}"),
                AIMessagePromptTemplate.from_template(suffix),
                MessagesPlaceholder(variable_name="agent_scratchpad"),
            ]
prompt = ChatPromptTemplate.from_messages(messages)

agent_executor = create_sql_agent(llm,
                                   toolkit=toolkit,
                                   agent_type="openai-tools",
                                   prompt = prompt,
                                   verbose=False)

Upvotes: 0

Related Questions