CODEWITHSUNDEEP
neo4jpy2neo
Aaron
Aaron

Reputation: 1743

Inserting large graph data into Neo4j using py2neo WriteBatch

I have a graph represented by the following files:

There are roughly 44K vertices with 240K edges. I'm trying to use neo4j.Writebatch to batch insert the graph data. 

from py2neo import Graph, neo4j, node, rel

graph_db = Graph()
nodes = {}
batchNodes = {}
edges = {}
edgeList = []

# Read vertex label file into nodes, where node[i] is indexed according to the order the nodes appear in the file.
# Each entry is of type node, e.g. node("FILM", title = "Star Trek"), node("CAST", name = "William Shatner")
...  

# Read edge label file into edges, where edges[i] is indexed according to the order the edges appear in the file.
# Each entry is a tuple (edge_type, edge_task), e.g. ("STAFF", "Director")
...  

# Read edge id file into edgeList
# Each entry is the tuple (source_index, target_index, edge_index), e.g. (1, 4, 8)
...  

# Iterate nodes, store in graph
# Note, store result of batch.create into batchNodes
batch = neo4j.WriteBatch(graph_db)
count = 0
for n in nodes:
    batchNodes[n] = batch.create(nodes[n])
    count += 1

    # Submit every 500 steps
    if count % 500 == 0:
        count = 0
        batch.submit()
        batch = neo4j.WriteBatch(graph_db)

# Submit remaining batch
batch.submit()

# Iterate edgeList, store in graph
batch = neo4j.WriteBatch(graph_db)
count = 0
for i, j, k in edgeList:
    # Lookup reference in batchNodes
    source = batchNodes[i]
    target = batchNodes[j]
    edge = edges[k]
    batch.create(rel(source, edge[0], target, {"task": edge[1]}))
    count += 1

    # Submit every 500 steps
    if count % 500 == 0:
        count = 0
        batch.submit()
        batch = neo4j.WriteBatch(graph_db)

# Submit remaining batch
batch.submit()

I get the following error:

Traceback (most recent call last):   File "test4.py", line 87, in <module>
    batch.create(rel(source, edge[0], target, {"task": edge[1]}))   File "C:\Python34\lib\site-packages\py2neo\batch\write.py", line 181, in create
    start_node = self.resolve(entity.start_node)   File "C:\Python34\lib\site-packages\py2neo\batch\core.py", line 374, in resolve
    return NodePointer(self.find(node))   File "C:\Python34\lib\site-packages\py2neo\batch\core.py", line 394, in find
    raise ValueError("Job not found in batch") ValueError: Job not found in batch

I presume that batchNodes is not actually containing the proper reference to the nodes which I want to lookup for adding relationships (perhaps reinitializing the batch object invalidates the references). In this case, how should I perform this task?

I am using Neo4j 2.1.7 (Community Edition) and py2neo 2.0.4.

Upvotes: 4

Views: 1519

Answers (1)

Michael Hunger
Michael Hunger

Reputation: 41676

For importing your CSV like data I'd recommend since Neo4j 2.1 LOAD CSV

load csv with headers from "file://...VertexLabel.txt" as row
where has(row.name)
create (:Actor {row.name})

similarly you can load your relationships

create index on :Actor(name); create index on :Movie(title);

load csv with headers from "file://...EdgeID.txt" as row
match (a:Actor {row.name})
match (m:Movie {row.title})
create (a)-[:ACTED_IN]->(m)

since Neo4j 2.2 you can also use neo4j-import a super fast tool to import csv data which also supports id-groups, providing labels and types in the csv etc.

see: http://neo4j.com/developer/guide-importing-data-and-etl/ and: http://neo4j.com/developer/guide-import-csv/

Upvotes: 2

Related Questions