CODEWITHSUNDEEP
javaspringhibernatespring-data-jpa
user6845744
user6845744

Reputation:

Spring boot - Jpa - SaveAll(Entity <S>) missing entities in the database

I am having an issue when I am trying to save a List of 21,300 entities, however in the database only [SELECT COUNT(id) = ] 10,506 are being saved.

The console output demonstrates the size of the list ready to be saved is correct so the code is "correct". If I make it loop only once with freq==30 for instance, this works fine so it seems linked somehow to the for each loop when it loops several times. The missing entities are somewhat dispersed in the population.

Any idea of why this particular behavior?

public void loadTickAll(int conid) {
    System.out.println(">>>> MERGING <<<<<< Querying ticks for " + conid);
    List<CandleTick> candlesHisto = histoCandleTickRepo.findByConidAll(conid, LocalDate.parse("2016-11-01"));
    List<CandleTick> candlesLive = new ArrayList<>();

    List<CandleTick> candles = Stream.of(candlesHisto, candlesLive).flatMap(Collection::stream)
            .collect(Collectors.toList());

     freqList.forEach((freq) -> {
     if(freq>=30) {
     System.out.println("Working on freq " + freq);

     FlowMerger flowMerger = new FlowMerger(freq, contracts.get(conid));
     List<CandleMerge> flow = flowMerger.createFlow(new ArrayList<>(candles));
     Collections.reverse(flow);

     System.out.println("Start saving " + freq + " - " + flow.size());
     histoCandleMergeRepo.saveAll(flow);

     System.out.println(freq + "/" + conid + " has been delivered and saved");
     }
     });

}

Console output

>>>> MERGING <<<<<< Querying ticks for 5 Working on freq 30 Start saving 30 - 21300 30/5 has been delivered and saved

application.properties

# jdbc.X
jdbc.driverClassName=org.postgresql.Driver

histo.jdbc.url=jdbc:postgresql://localhost:5432/trading2018
live.jdbc.url=jdbc:postgresql://localhost:5433/trading2018

jdbc.user=...
jdbc.pass=...


## hibernate.X
hibernate.default_schema=trading
hibernate.jdbc.lob.non_contextual_creation=true
hibernate.show_sql=false
hibernate.temp.use_jdbc_metadata_defaults=false
hibernate.dialect=org.hibernate.dialect.PostgreSQLDialect
hibernate.hbm2ddl.auto=validate
#hibernate.cache.use_second_level_cache=false
#hibernate.cache.use_query_cache=false

logging.level.org.springframework.web=ERROR
logging.level.com=DEBUG
logging.file=C:/tmp/application.log

Upvotes: 0

Views: 2563

Answers (1)

user6845744
user6845744

Reputation:

I managed to trace down the issue. I showed the SQL and noticed I was updating instead of inserting.

My entity was missing some annotations...

@GeneratedValue(generator = "merge_id_seq")
@SequenceGenerator(name = "candleid_generator_generator", sequenceName = "merge_id_seq", initialValue = 1)
@Id
private int id;

Upvotes: 1

Related Questions