CODEWITHSUNDEEP
fastapillama-indexollamallama3
Sun Bee
Sun Bee

Reputation: 1820

Llama-index chatbot in FastAPI endpoint throws "Event loop is already running"

The end-point implementation is like so:

@app.post("/api/chat/{question}", dependencies=[Depends(sessionValidator)])
async def chat(question: str = Path(...), my_chatbot=Depends(chatbotProvider)):
    # Generate response
    try:
        logger.info(f"Getting LLM response..")

        ans = my_chatbot.chatbot.chat(question)
        
        logger.info(f"SUCCESS! Chatbot responded: {ans.response}")

        return ans.response
    
    except Exception as e:
        logger.error(f"FAILED! Got no response from chatbot! Err {e}")
        raise HTTPException(status_code=500, detail="Chatbot gave no response!")

The chatbot (whose chat method I have called) is llama-index chat engine based on vector index. It is an instance of class BaseChatEngine from llama_index.core.chat_engine.types and uses an Ollama Lllama3 model from llama_index.llms.ollama.

Here is the code the creates the chatbot in my wrapper class MyChatAssistant that has the llama-index's BaseChatEngine instance as attribute chatbot:

self.chatbot = self.index.as_chat_engine(
                chat_mode = self._chat_mode,
                verbose = True,
                similarity_top_k = 6,
                filters = MetadataFilters(filters=[
                    ExactMatchFilter(key="transcript_id", value=self.transcript_id),]        
                )
            )

Interestingly, I have extensively tested the chatbot (minus Rest API endpoint wrapper) in Jupyter notebook. There I used:

import nest_asyncio
nest_asyncio.apply()

This took care of all issues. The same did not work in FastAPI as it seems the event loop that FastAPI uses is a different type.

I have tried the following:

4.

ans = asyncio.get_event_loop().run_until_complete(my_chatbot.chatbot.chat(question))
# or
loop = asyncio.get_event_loop()
ans = loop.create_task(my_chatbot.chatbot.chat(question))
await and

Err: This event loop is alreay running

3.

ans = asyncio.to_thread(my_chatbot.chatbot.chat(question))
# or
ans = await asyncio.to_thread(my_chatbot.chatbot.chat(question))

Err: This event loop is alreay running

2.

loop = asyncio.get_event_loop()
ans = await loop.run_in_executor(None, my_chatbot.chatbot.chat, question)

ans = my_chatbot.chatbot.chat(question)
# or
ans = await my_chatbot.chatbot.chat(question)

Grateful for any help!

Upvotes: 0

Views: 50

Answers (1)

Yurii Motov
Yurii Motov

Reputation: 2323

I guess my_chatbot.chatbot.chat() is a sync wrapper that runs async function in the event loop. So, you need to use async alternative of this method.

Try changing

        ans = my_chatbot.chatbot.chat(question)

to

        ans = await my_chatbot.chatbot.achat(question)

Upvotes: 0

Related Questions