VanechikSpace
VanechikSpace

Reputation: 555

Is llama able to choose tools automatically?

I run the following code expecting llama will decide whether to use the tool or not depending on my prompt:

from llama_cpp import Llama,ChatCompletionNamedToolChoice

llm = Llama(
  model_path="/home/s1ngle/.cache/huggingface/hub/models--bartowski--Meta-Llama-3.1-8B-Instruct-GGUF/snapshots/9a8dec50f04fa8fad1dc1e7bc20a84a512e2bb01/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf",

  chat_format="chatml-function-calling",

  verbose=False
)

result = llm.create_chat_completion(
  messages = [
    {"role": "user", "content": "Hello, what is the wheather ?"},
    {"role": "user", "content": "Hello !"}
  ],

  tools=[{
    "type": "function",
    "function": {
        "name": "OUYEAH",
        "description": "used whenever the model is being asked to find out the current weather",
    }
  }],

  tool_choice="auto",
)

print(result)

The output is:

{'id': 'chatcmpl-bfdc0dc7-6b16-435f-bd89-4ab644a823be', 'object': 'chat.completion', 'created': 1727433781, 'model': '/home/s1ngle/.cache/huggingface/hub/models--bartowski--Meta-Llama-3.1-8B-Instruct-GGUF/snapshots/9a8dec50f04fa8fad1dc1e7bc20a84a512e2bb01/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': ''}, 'logprobs': None, 'finish_reason': 'stop'}], 'usage': {'prompt_tokens': 49, 'completion_tokens': 6, 'total_tokens': 55}

Upvotes: 1

Views: 163

Answers (0)

Related Questions