CODEWITHSUNDEEP
springlarge-language-modelllamaspring-ai
batmaniac
batmaniac

Reputation: 362

RAG spring AI on simple database table not finding data with llama3 model

When I try to use spring AI to query a table using ollama3 model I get bad results. It seems my data is not used.

The table "Accedant" is very simple. It contains a name, firstname and email (in french language).

All the code is here in this package src/main/java/com/sivalabs/aidemo/ragdb of this repository https://github.com/farous/ai.git

For example I ask this question in the browser : http://localhost:8080/api/query?question=combien d'accedants ont le prenom ADRIEN

It answers that there are no accedant named ADRIEN in the table, which is false.

Thank you very much for your help.

Upvotes: 1

Views: 321

Answers (2)

Jens Schauder
Jens Schauder

Reputation: 81990

What you are essentially doing is the following:

  • get a question from the user.
  • concatenate it with the content of your database and some template text.
  • send that to your AI model.

The problem here is that the resulting query is way to long. According to comments, the model you are using supports 8K tokens. with 15K entries in your database, thats already twice the amount of tokens when each entry would be encoded in a single toke. But it's probably more in the order tens or hundreds of tokens per entry.

While this could be solved with a model supporting a bigger context, this approach wouldn't scale and be unnecessary expensive (in $ or compute or both).

The correct way to do RAG is to store your data in a vector store. Then when you get the query, you will retrieve the N most relevant entries from your database and only include those in the query.

Your use case might also lend itself to a different approach where you register various database queries as functions, so the model can actually produce exact answers for questions that can be answered by one of these queries.

Upvotes: 0

Pradip
Pradip

Reputation: 133

I tried a similar excercise with llama3.1:8b on "Indian General Election 2024" data and got the exact same problem, for example when I asked "How many votes Narendra Modi got?" it said there's no data about him which was obviously false.

Based on my experience on other tabular data as well, it seems like llama3 doesn't work well with tabular data especially if the number of rows are large, the major reason being context size. We need to chunk our input data for embeddings in batches and if we try too large a chunk size to retain more context it will throw errors like - "cannot submit more than xyz embeddings at once", "batch size abc exceeds maximum batch size xyz".

This is a common problem with free models, you can try some paid models (which supports larger context size) by purchasing API token keys which give better results. I tried with GPT3.5 on OpenAI and it gave better results. However, as mentioned earlier, for larger tabular data LLM models struggle to give accurate answers.

Hope this helps.

Upvotes: 1

Related Questions