Reputation: 555
I run the following code expecting llama will decide whether to use the tool or not depending on my prompt:
from llama_cpp import Llama,ChatCompletionNamedToolChoice
llm = Llama(
model_path="/home/s1ngle/.cache/huggingface/hub/models--bartowski--Meta-Llama-3.1-8B-Instruct-GGUF/snapshots/9a8dec50f04fa8fad1dc1e7bc20a84a512e2bb01/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf",
chat_format="chatml-function-calling",
verbose=False
)
result = llm.create_chat_completion(
messages = [
{"role": "user", "content": "Hello, what is the wheather ?"},
{"role": "user", "content": "Hello !"}
],
tools=[{
"type": "function",
"function": {
"name": "OUYEAH",
"description": "used whenever the model is being asked to find out the current weather",
}
}],
tool_choice="auto",
)
print(result)
The output is:
{'id': 'chatcmpl-bfdc0dc7-6b16-435f-bd89-4ab644a823be', 'object': 'chat.completion', 'created': 1727433781, 'model': '/home/s1ngle/.cache/huggingface/hub/models--bartowski--Meta-Llama-3.1-8B-Instruct-GGUF/snapshots/9a8dec50f04fa8fad1dc1e7bc20a84a512e2bb01/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'content': ''}, 'logprobs': None, 'finish_reason': 'stop'}], 'usage': {'prompt_tokens': 49, 'completion_tokens': 6, 'total_tokens': 55}
Upvotes: 1
Views: 163