CODEWITHSUNDEEP
springneo4jspring-data-neo4j
Stéphane Traumat
Stéphane Traumat

Reputation: 119

neo4j slows down after lots of inserts

I'm the owner of the Blockchain2graph project that reads data from Bitcoin core rest API and insert Blocks, Addresses and Transactions as Graph objects in Neo4j.

After some imports, the process is slowing down until the memory is full. I don't want to use CSV imports. My problem is not performance, my goal is to insert things without the application stopping because of memory (even if it takes quite a lot of time)

I'm using spring-boot-starter-data-neo4j.

In my code, I try to make session.clear from times to times but it doesn't seem to have an impact. After restarting tomcat8, things go fast again.

Upvotes: 0

Views: 341

Answers (1)

Michael Hunger
Michael Hunger

Reputation: 41676

As your project is about mass inserts, I wouldn't use an OGM like Spring Data Neo4j for writing the data.

You don't want a session to keep your data around on the client.

Instead, use Cypher directly sending updates you get from the BlockChain API directly as a batch per request, see my blog post for some examples (some of which we also use in SDN/Neo4j-OGM under the hood).

You can still use SDN for individual entity handling (CRUD) that's what OGMs are good for in my book to reduce the boilerplate.

But for more complex read operations that have aggregation, filtering, projection and path matches I'd still use Cypher on an annotated repository method, returning rows that can be mapped to a list of DTOs.

Upvotes: 2

Related Questions