CODEWITHSUNDEEP
pythonllama-index
Vivek
Vivek

Reputation: 276

How can we add a list of documents to an existing index in llama-index?

I have an existing index that is created using GPTVectorStoreIndex. However, when I am trying to add a new document to the existing index using the insert method, I am getting the following error :

AttributeError: 'list' object has no attribute 'get_text'

my code for updating the index is as follows :

max_input_size = 4096
num_outputs = 5000
max_chunk_overlap = 256
chunk_size_limit = 3900
prompt_helper = PromptHelper(max_input_size, num_outputs, max_chunk_overlap, chunk_size_limit=chunk_size_limit)
llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name="gpt-3.5-turbo", max_tokens=num_outputs))
    
service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor, prompt_helper=prompt_helper)

directory_path = "./trial_docs"
file_metadata = lambda x : {"filename": x}
reader = SimpleDirectoryReader(directory_path, file_metadata=file_metadata)
    
documents = reader.load_data()
print(type(documents))
index.insert(document = documents, service_context = service_context)

Upvotes: 4

Views: 5737

Answers (1)

Vivek
Vivek

Reputation: 276

I got it right, the mistake I was doing it was passing documents as a whole, which is a List object. The right way to update is as follows

max_input_size = 4096
num_outputs = 5000
max_chunk_overlap = 256
chunk_size_limit = 3900
prompt_helper = PromptHelper(max_input_size, num_outputs, max_chunk_overlap, chunk_size_limit=chunk_size_limit)
llm_predictor = LLMPredictor(llm=OpenAI(temperature=0, model_name="gpt-3.5-turbo", max_tokens=num_outputs))
    
service_context = ServiceContext.from_defaults(llm_predictor=llm_predictor, prompt_helper=prompt_helper)

directory_path = "./trial_docs"
file_metadata = lambda x : {"filename": x}
reader = SimpleDirectoryReader(directory_path, file_metadata=file_metadata)
    
documents = reader.load_data()
print(type(documents))
for d in documents:
    index.insert(document = d, service_context = service_context)

Upvotes: 8

Related Questions