CODEWITHSUNDEEP
neo4j
Bilal and Olga
Bilal and Olga

Reputation: 3251

Windows Neo4j cannot create graph with 25000 nodes due to memory error

I am trying to create a graph on a Windows machine with 350,000 nodes - unfortunately i cannot even create one with 25,000 nodes. 500 nodes works. With 25,000 it runs for 20 mins and then gives a Java heap error.

Below is the cypher script and in the csv file i control the number of rows (would like to do 350K). Error I get is Java heap - i tried increasing memory config to 1024, didn't help and it wouldn't let me increase to 2048.

LOAD CSV WITH HEADERS FROM "file:c:...my file.csv" AS csvLine
MERGE (f:F { name: csvLine.f })
MERGE (s:S { name: csvLine.s })
MERGE (t:T { name: csvLine.t })
MERGE (region:R { name: csvLine.r })
CREATE (d:D { name: csvLine.name})
CREATE (d)-[:BELONGS_TO]->(f)
CREATE (f)-[:BELONGS_TO]->(r)
CREATE (d)-[:LOCATED_IN]->(s)
CREATE (d)-[:LOCATED_IN]->(t)
CREATE (s)-[:PINGS{c: toInt(csvLine.c)}]->(t)

Upvotes: 0

Views: 83

Answers (1)

Michael Hunger
Michael Hunger

Reputation: 41686

Unfortunately you have to do three things

  1. Update to Neo4j 2.1.3

  2. Create the necessary indexes

  3. Simplify your statements into multiple simpler ones. The reason being that Cypher (when you match + create data) has to pull in all merge/match statemets in upfront to be not affected by data created later on.

 USING PERIODIC COMMIT 1000
 LOAD CSV WITH HEADERS FROM "file:c:...my file.csv" AS csvLine
 MERGE (f:F { name: csvLine.f });


USING PERIODIC COMMIT 1000
LOAD CSV WITH HEADERS FROM "file:c:...my file.csv" AS csvLine
MERGE (s:S { name: csvLine.s });

USING PERIODIC COMMIT 1000
LOAD CSV WITH HEADERS FROM "file:c:...my file.csv" AS csvLine
MERGE (t:T { name: csvLine.t });

USING PERIODIC COMMIT 1000
LOAD CSV WITH HEADERS FROM "file:c:...my file.csv" AS csvLine
MERGE (region:R { name: csvLine.r });

USING PERIODIC COMMIT 1000
LOAD CSV WITH HEADERS FROM "file:c:...my file.csv" AS csvLine
CREATE (d:D { name: csvLine.name});

USING PERIODIC COMMIT 1000
LOAD CSV WITH HEADERS FROM "file:c:...my file.csv" AS csvLine
MATCH (f:F { name: csvLine.f })
MATCH (s:S { name: csvLine.s })
MATCH (t:T { name: csvLine.t })
MATCH (region:R { name: csvLine.r })
MATCH (d:D { name: csvLine.name})
CREATE (d)-[:BELONGS_TO]->(f)
CREATE (f)-[:BELONGS_TO]->(r)
CREATE (d)-[:LOCATED_IN]->(s)
CREATE (d)-[:LOCATED_IN]->(t)
CREATE (s)-[:PINGS{c: toInt(csvLine.c)}]->(t);

For the first ones you can also use this construct if you only import a single file (these will be faster)

USING PERIODIC COMMIT 1000
LOAD CSV WITH HEADERS FROM "file:c:...my file.csv" AS csvLine
WITH distinct csvLine.f as csv_f
CREATE (f:F { name: csv_f });

etc.

Upvotes: 1

Related Questions