CODEWITHSUNDEEP
openai-apilarge-language-modelmistral-7b
Botellita Taponzito
Botellita Taponzito

Reputation: 121

Mistral does not finish the answers

I am developing a web application to be able to answer questions based on the context provided by documents that the user uploads to the application. The problem is that when I use the Mistral v0.2 model, the answers do not finish. They are cut off before finishing. If I use openai, the answers finish correctly. I use this prompt:

template="""
    ### [INST] Instruccion: Responde en español a las preguntas del usuario según el contexto.
    Si no encunetras una respuesta adecuada en el contexto, responde que no tienes información suficiente.

    {context}

    ### question:
    {question} (responde en castellano) [/INST]
    #"""
    template="""
    <s>[INST]
    """
prompt = PromptTemplate(
        input_variables=['context','question'],
        template = template
    )
vector = Chroma(client=db,
        collection_name="coleccion4",
        embedding_function=embeddings)
retriever = vector.as_retriever(search_type="similarity", search_kwargs={"k":3})
llm = HuggingFaceHub(
        repo_id="mistralai/Mistral-7B-Instruct-v0.2",
        model_kwargs = {"temperature":0.4},
        huggingfacehub_api_token = apikey_huggingFace
    )
respuesta = rag_chain.invoke(user_question)

when I run the code with openai, I get this response: enter image description here

But when I use Mistral, the answer does not end: enter image description here

why does this happen?

Upvotes: 0

Views: 1217

Answers (2)

Botellita Taponzito
Botellita Taponzito

Reputation: 121

I have set max_new_tokens to 2000 and now it seems to work

Upvotes: -1

Pratyush Vel Shankar
Pratyush Vel Shankar

Reputation: 1

There are multiple issues that could be going on here -

  1. Ram Limit Reached: Please make sure you have more that 8.0 ram usable
  2. Temparature - 0.4 is very specific and may cause short responses
  3. Token Limit - Try specifying a token limit to the desired lenght of your response.

Upvotes: 0

Related Questions