Does langchain with llama-cpp-python fail to work with very long prompts?

Question

I'm trying to create a service using the llama3-70b model by combining langchain and llama-cpp-python on a server workstation. While the model works well with short prompts(question1, question2), it produces unknown words for very long prompts(question3)(though smaller than 8192 tokens). This issue is also present with the 8b model.

For example, "a a a the the the / / / ( and / / / <> ....."

I haven't found any cases of people experiencing the same problem through my searches. I also installed and tested on Ollama with same prompts, but it worked well.

Below are my python code and specs of GPUs.

from langchain_community.llms import LlamaCpp
from langchain_core.prompts import PromptTemplate
from langchain_core.callbacks.manager import CallbackManager
from langchain_core.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain_core.output_parsers import StrOutputParser

import multiprocessing

question_system = """
You know all knowledges of over the world. Answer the user's question.
"""

question1 = """
Question: Show me a rap battle between Stephen Colbert and John Oliver. And traslate it to french of your answer. Also show me english and french versions both. I think it is better that the battle is longer.

Answer:
"""

question2 = """
Question: Make longest questions that you can be. I want to prove your ability, so make 20 extremely long questions as you as possible.

Answer: 
"""

question3 = """
Question List:

1. Can a group of highly skilled and extensively trained experts in the field of artificial intelligence, with access to vast amounts of data and computational resources, successfully develop an AI system that can accurately predict and prevent all types of cyber attacks, including those that exploit zero-day vulnerabilities and use advanced evasion techniques?

2. Would it be possible for a team of experienced software developers, working together with a group of skilled security experts, to design and implement a secure and reliable operating system, one that is capable of protecting sensitive data and preventing malicious code from executing, even in the presence of sophisticated attacks and exploits?

3. Can a group of researchers and scientists, working together across multiple disciplines, including computer science, biology, chemistry, and physics, successfully develop a new generation of advanced nanotechnology-based systems, ones that are capable of operating at the molecular level, with precision and control, to perform complex tasks, such as medical imaging, drug delivery, and tissue engineering?

4. Would it be possible for a team of skilled engineers and technicians, working together with a group of experienced scientists and researchers, to design and develop a new generation of advanced spacecraft, ones that are capable of traveling to other planets and celestial bodies in our solar system, with precision and control, to perform complex tasks, such as planetary exploration, resource utilization, and astrobiology research?

5. Can a group of highly skilled and extensively trained experts in the field of cybersecurity, with access to vast amounts of data and computational resources, successfully develop an AI-powered system that can accurately detect and prevent all types of cyber attacks, including those that exploit zero-day vulnerabilities and use advanced evasion techniques?

6. Would it be possible for a team of experienced software developers, working together with a group of skilled security experts, to design and implement a secure and reliable operating system, one that is capable of protecting sensitive data and preventing malicious code from executing, even in the presence of sophisticated attacks and exploits?

7. Can a group of researchers and scientists, working together across multiple disciplines, including computer science, biology, chemistry, and physics, successfully develop a new generation of advanced nanotechnology-based systems, ones that are capable of operating at the molecular level, with precision and control, to perform complex tasks, such as medical imaging, drug delivery, and tissue engineering?

8. Would it be possible for a team of skilled engineers and technicians, working together with a group of experienced scientists and researchers, to design and develop a new generation of advanced spacecraft, ones that are capable of traveling to other planets and celestial bodies in our solar system, with precision and control, to perform complex tasks, such as planetary exploration, resource utilization, and astrobiology research?

9. Can a group of highly skilled and extensively trained experts in the field of artificial intelligence, with access to vast amounts of data and computational resources, successfully develop an AI-powered system that can accurately predict and prevent all types of natural disasters, including earthquakes, hurricanes, wildfires, and floods?

10. Would it be possible for a team of experienced software developers, working together with a group of skilled security experts, to design and implement a secure and reliable operating system, one that is capable of protecting sensitive data and preventing malicious code from executing, even in the presence of sophisticated attacks and exploits?

11. Can a group of researchers and scientists, working together across multiple disciplines, including computer science, biology, chemistry, and physics, successfully develop a new generation of advanced nanotechnology-based systems, ones that are capable of operating at the molecular level, with precision and control, to perform complex tasks, such as medical imaging, drug delivery, and tissue engineering?

12. Would it be possible for a team of skilled engineers and technicians, working together with a group of experienced scientists and researchers, to design and develop a new generation of advanced spacecraft, ones that are capable of traveling to other planets and celestial bodies in our solar system, with precision and control, to perform complex tasks, such as planetary exploration, resource utilization, medical imaging, drug delivery, and asteroid

13. Can a group of highly skilled and extensively trained experts in the field of artificial intelligence, with access to vast amounts of data and computational resources, successfully develop an AI system that can accurately predict and prevent all types of cyber attacks, including those that exploit zero-day vulnerabilities and use advanced evasion techniques?

14. Would it be possible for a team of experienced software developers, working together with a group of skilled security experts, to design and implement a secure and reliable operating system, one that is capable of protecting sensitive data and preventing malicious code from executing, even in the presence of sophisticated attacks and exploits?

15. Can a group of researchers and scientists, working together across multiple disciplines, including computer science, biology, chemistry, and physics, successfully develop a new generation of advanced nanotechnology-based systems, ones that are capable of operating at the molecular level, with precision and control, to perform complex tasks, such as medical imaging, drug delivery, and tissue engineering?

16. Would it be possible for a team of skilled engineers and technicians, working together with a group of experienced scientists and researchers, to design and develop a new generation of advanced spacecraft, ones that are capable of traveling to other planets and celestial bodies in our solar system, with precision and control, to perform complex tasks, such as planetary exploration, resource utilization, and astrobiology research?

17. Can a group of highly skilled and extensively trained experts in the field of cybersecurity, with access to vast amounts of data and computational resources, successfully develop an AI-powered system that can accurately detect and prevent all types of cyber attacks, including those that exploit zero-day vulnerabilities and use advanced evasion techniques?

18. Would it be possible for a team of experienced software developers, working together with a group of skilled security experts, to design and implement a secure and reliable operating system, one that is capable of protecting sensitive data and preventing malicious code from executing, even in the presence of sophisticated attacks and exploits?

19. Can a group of researchers and scientists, working together across multiple disciplines, including computer science, biology, chemistry, and physics, successfully develop a new generation of advanced nanotechnology-based systems, ones that are capable of operating at the molecular level, with precision and control, to perform complex tasks, such as medical imaging, drug delivery, and tissue engineering?

20. Would it be possible for a team of skilled engineers and technicians, working together with a group of experienced scientists and researchers, to design and develop a new generation of advanced spacecraft, ones that are capable of traveling to other planets and celestial bodies in our solar system, with precision and control, to perform complex tasks, such as planetary exploration, resource utilization, and astrobiology research?

21. Can a group of highly skilled and extensively trained experts in the field of artificial intelligence, with access to vast amounts of data and computational resources, successfully develop an AI-powered system that can accurately predict and prevent all types of natural disasters, including earthquakes, hurricanes, wildfires, and floods?

22. Would it be possible for a team of experienced software developers, working together with a group of skilled security experts, to design and implement a secure and reliable operating system, one that is capable of protecting sensitive data and preventing malicious code from executing, even in the presence of sophisticated attacks and exploits?

23. Can a group of researchers and scientists, working together across multiple disciplines, including computer science, biology, chemistry, and physics, successfully develop a new generation of advanced nanotechnology-based systems, ones that are capable of operating at the molecular level, with precision and control, to perform complex tasks, such as medical imaging, drug delivery, and tissue engineering?

24. Would it be possible for a team of skilled engineers and technicians, working together with a group of experienced scientists and researchers, to design and develop a new generation of advanced spacecraft, ones that are capable of traveling to other planets and celestial bodies in our solar system, with precision and control, to perform complex tasks, such as planetary exploration, resource utilization, medical imaging, drug delivery, and asteroid?

Question: Answer about the above questions shortly, and count the questions.

Answer: 
"""

template = """
{System}<|start_header_id|>system<|end_header_id|>

{System}<|eot_id|>

{Prompt}<|start_header_id|>user<|end_header_id|>

{Prompt}<|eot_id|>

<|start_header_id|>assistant<|end_header_id|>

{Response}<|eot_id|>
"""

prompt = PromptTemplate.from_template(template)

llm = LlamaCpp(
    model_path="/HDD/llama3/Meta-Llama-3-70B-Instruct.Q4_K_M.gguf",
    n_gpu_layers=-1,
    n_ctx=8192,
    n_batch=128,
    max_tokens=1024,
    callback_manager=CallbackManager([StreamingStdOutCallbackHandler()]),
    verbose=True,
    stop=["<|start_header_id|>", "<|end_header_id|>", "<|eot_id|>"],
    n_threads=multiprocessing.cpu_count()-1,
)

chain = prompt | llm | StrOutputParser()
res = chain.invoke({"System":question_system, "Prompt":question3, "Response":None})
print(res)

specs of workstation

• OS : Ubuntu 20.04
• GPUs : 6 RTX 8000 (VRAM : 6 x 48Gb)
• langchain : v0.2.8
• llama-cpp-python : v0.2.82
• model file : Meta-Llama-3-70B-Instruct.Q4_K_M.gguf

What I tried....

1. I changed my configures, such as increasing n_ctx to 8192 * 10 and n_batch to 128 * 10, decreasing n_gpu_layers to 1,2,4 but it had same problem.
2. I removed n_threads, but it's too.
3. I tried using ChatLlamaCpp, but it's too.
4. I can't understand why the same prompts works on Ollama, but doesn't work on my code.

Answer 1

It seems like your prompt is well below the 8192 token context length set as n_ctx=8192, yet your still limiting the LLM response to 1024 tokens set as max_tokens=1024,.

Try making max_tokens = n_ctx - <input tokens> where you'd calculate <input_tokens> based on your prompt. This should give the LLM more room to answer the 24 questions.

Considering that 'question 3' is really a list of 24 questions and the task is to answer all of them, the response may very well exceed the the 1024 tokens you set as an upper limit to the response. When this happens, there is still some room in your context length so the LLM may be supplying some garbage.

As a note, increasing n_ctx beyond the context window with which the model was trained on is not recommended, so setting n_ctx to 8192 * 10 will do more harm than good.