CODEWITHSUNDEEP
pythonstreamlitreplicatellamallama-index
anikaM
anikaM

Reputation: 429

replicate with llama-index and streamlit - unable to find documents

I am running code below on a GPU via:

streamlit run replicate_lama2.py

import os
import traceback
import sys
import streamlit as st

os.environ["REPLICATE_API_TOKEN"] = "my_key"

from llama_index.llms import Replicate

llama2_7b_chat = "meta/llama-2-7b-chat:8e6975e5ed6174911a6ff3d60540dfd4844201974602551e10e9e87ab143d81e"
llm = Replicate(
    model=llama2_7b_chat,
    temperature=0.01,
    additional_kwargs={"top_p": 1, "max_new_tokens": 300},
)

from llama_index import VectorStoreIndex, SimpleDirectoryReader
from llama_index.embeddings import HuggingFaceEmbedding
from llama_index import ServiceContext

embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
service_context = ServiceContext.from_defaults(
    llm=llm, embed_model=embed_model
)

documents = SimpleDirectoryReader("data").load_data()
index = VectorStoreIndex.from_documents(
    documents, service_context=service_context
)


#index = VectorStoreIndex.from_documents(documents)

# Get the query engine
query_engine = index.as_query_engine(streaming=True)

# Create a Streamlit web app
#st.title("LLM Query Interface")
query = st.text_input("Enter your query:")
submit_button = st.button("Submit")

if submit_button:
    # Query the engine with the defined query
    response = query_engine.query(query)
    st.write("### Query Result:")
    st.write(response)

My directory "data" that contains different files I want to perform my queries is in the same directory where replicate_lama2.py script it. When I run this I am getting streamlit to open this chat in web browser (firefox) and I can ask a question (which is definitely answerable from documents in /data dir) but instead of any answer I am getting output in attach, that says "No doc available". How to make this work?enter image description here

Upvotes: 0

Views: 176

Answers (1)

Ankit
Ankit

Reputation: 21

The object type is - class llama_index.core.base.response.schema.StreamingResponse(response_gen: ~typing.Generator[str, None, None], source_nodes: ~typing.List[~llama_index.core.schema.NodeWithScore] = , metadata: ~typing.Optional[~typing.Dict[str, ~typing.Any]] = None, response_txt: ~typing.Optional[str] = None)

so use instead - st.write(response.response)

Upvotes: 0

Related Questions