CODEWITHSUNDEEP
openai-apilangchainllama-index
Yousif Abdalla
Yousif Abdalla

Reputation: 1587

llama-index: How to Extract `context_str` Used in LlamaIndex for a Given Response?

I'm working with LlamaIndex, and I need to extract the context_str that was used in a query before it was sent to the LLM (Language Model). Here's the relevant code:

index = VectorStoreIndex.from_documents(
    documents, storage_context=storage_context, service_context=service_context
)
query_engine = index.as_query_engine()

response = query_engine.query("was July 2023 recorded as the hottest day on Earth?")
display(Markdown(f"<b>{response}</b>"))

The output of this code is as follows:

July 4, 2023 was recorded as the hottest day on Earth since at least 1979, according to data from the U.S. National Centers for Environmental Prediction. Some scientists believe it may have been one of the hottest days on Earth in about 125,000 years.

I understand that the Llama query sent a similar prompt like this to LLM:

prompt(
    "We have provided context information below. \n"
    "---------------------\n"
    "{context_str}"
    "\n---------------------\n"
    "Do not give me an answer if it is not mentioned in the context as a fact. \n"
    "Given this information, please provide me with an answer to the following:\n{query_str}\n"
)

how can I extract the context_str from the prompt that was used to generate the response?

Upvotes: 5

Views: 4859

Answers (3)

Drew Galbraith
Drew Galbraith

Reputation: 563

Prose Answer

(tl;dr, there is a code answer below)

The response from both index.as_query_engine().query() or index.as_retriever.retrieve() (suggested here) will be an object from the Response class, as noted in @AalaaNagy answer:

query_engine = index.as_query_engine()
response = query_engine.query("was July 2023 recorded as the hottest day on Earth?")

print(type(response))  # Output:  <class 'llama_index.core.base.response.schema.Response'>

If we run dir(response), we can see the properties of this object. I usually skip all the double-underscore methods/attributes, and skip straight to the other stuff:

['__annotations__', '__class__', ..., 'get_formatted_sources', 'metadata', 'response', 'source_nodes']

And from here one can just print out the most subjectively promising methods/attributes (e.g., print(response.source_nodes)), and get a good understanding of the Response object from there. I found the following code worked to get context from llama-index Response objects.

Code Answer

query_engine = index.as_query_engine() # Optionally, use 'similarity_top_k' param here
response = query_engine.query("<your_query>")

context = " ".join([node.dict()['node']['text'] for node in response.source_nodes])
print(context)

Upvotes: 1

Aalaa Nagy
Aalaa Nagy

Reputation: 1

Just print the response and you will find all details you need like context_str and page_number

The response is an instance from class Response and you can access the nodes from print(response.sourcenodes)

Upvotes: -1

ZKS
ZKS

Reputation: 2836

Reference https://github.com/jerryjliu/llama_index/issues/1045. You can create a custom class extending LLM, you need to slightly modify so that you can make call to LLM. This code does not make call to LLM, however you can modify it based on you requirement

    from langchain.llms.base import LLM
    from typing import Optional, List, Mapping, Any

    class CustomLLM(LLM):
        def _call(self, prompt: str, stop: Optional[List[str]] = None) -> str:
            return prompt

        @property
        def _identifying_params(self) -> Mapping[str, Any]:
            return {"name_of_model": "echo"}

        @property
        def _llm_type(self) -> str:
            return "custom"

    from llama_index import LLMPredictor
    llm_predictor = LLMPredictor(llm=CustomLLM())

    model_name="x"
    embed_model = LangchainEmbedding(HuggingFaceEmbeddings(model_name=model_name))
    service_context = ServiceContext.from_defaults(embed_model=embed_model,llm_predictor=llm_predictor)

Upvotes: 0

Related Questions