CODEWITHSUNDEEP
neo4j
user3515307
user3515307

Reputation: 11

Neo4j 2.1.0-M01 fails badly with tree graph of 8,3 millions leaves

(already opened a case #2250 on Github but maybe someone here has a solution?) he story. We have a dataset which is just a tree with

We are loading data with LOAD CSV where each line creates a node and a relationship to a previous level node. Neo4j is a 2.1.0-M01 Enterprise-for-Startups, cluster with 3 nodes, 8Gb memory at each instance, JVM is told to use 7Gb. Part I. We discovered that at levels 4 and 5 it takes some considerable (minutes) time until loaded level becomes visible from Cypher queries. We wrote a simple script which just dumps all possible paths in a tree:

START root = node(0) match path = (root)-[r*]->(b)
WITH length(path) as len, path
RETURN len, extract( n IN NODES(path) | n.payload ) AS payload_chain
ORDER BY len DESC

You can easily guess that it returns the quantity of path exactly equal to the quantity of relationships in database.

Immediately after LOAD CSV completion at level 4, web system monitor says that we have 49432 relationships and 49433 nodes which is what expected. But script above reports some 26k+ paths in the first immediate run, some 35k+ in second run and step by step it shows exactly what was expected: 49432. Ok so far, but we are USING PERIODIC COMMIT 5000 so supposedly all the transactions should be commited before the first run of the script? The same story was at level 5: first run gave 223641 paths, second 293872, third 294482, fourth 476334, fifth gave 794482 which is what was intended. Anyway we concluded that script is a good diagnostic tool to make sure that prevoius level is complete before starting import of the next level.

Part II. We are loading 8340k+ nodes to level 6 (it takes some time). Import wents fine with no errors, nothing bad to suspect. Than we start our test script and here the horror begins. The first attempt to run the script cause master instance to fail (log is atteched below). Cluster told us that one of the slaves has become master, and didn't signal any errors. But all of a sudden, at the next run our script returned... 0 (zero!) paths immediately. Query which counts relationships (root)-[r]->() returns correct value 249. Ok I tried to change the script to

MATCH path = (root:MyLabel {payload: 'root_payload'})-[r*]->(b)
WITH length(path) as len, path
RETURN len, extract( n IN NODES(path) | n.payload ) AS payload_chain
ORDER BY len DESC

it also returned 0 (zero) but after some time of thinking.

So we are stuck with (supposedly) corrupted database - at restart, instances say that datastores are Ok and consistent - and we are unable to complete this simple import of the tree into Neo4j. What are we doing wrong? Or maybe it is a known issue which is already solved in M02?

Would you mind suggesting us some solution, please?

Thanks in advance! WBR, Konstantin

Ukraine

------ 8< log follows 8< -----

13:25:41.361 [qtp1312809772-7217] WARN o.e.jetty.servlet.ServletHandler - /db/manage/server/monitor/fetch javax.ws.rs.WebApplicationException: javax.ws.rs.WebApplicationException: org.eclipse.jetty.io.EofException at org.neo4j.server.rest.repr.OutputFormat$1.write(OutputFormat.java:178) ~[neo4j-server-2.1.0-M01.jar:2.1.0-M01] at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71) ~[jersey-core-1.9.jar:1.9] at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57) ~[jersey-core-1.9.jar:1.9] at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) ~[jersey-server-1.9.jar:1.9] at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) ~[jersey-server-1.9.jar:1.9] at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) ~[javax.servlet-3.0.0.v201112011016.jar:na] at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:698) ~[jetty-servlet-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1506) ~[jetty-servlet-9.0.5.v20130815.jar:9.0.5.v20130815] at org.neo4j.server.rest.security.SecurityFilter.doFilter(SecurityFilter.java:112) ~[neo4j-server-2.1.0-M01.jar:2.1.0-M01] at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1477) ~[jetty-servlet-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503) [jetty-servlet-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:211) [jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1096) [jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:432) [jetty-servlet-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175) [jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1030) [jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136) [jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) [jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.server.Server.handle(Server.java:445) [jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:268) [jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:229) [jetty-server-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358) [jetty-io-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:601) [jetty-util-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:532) [jetty-util-9.0.5.v20130815.jar:9.0.5.v20130815] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51] Caused by: javax.ws.rs.WebApplicationException: org.eclipse.jetty.io.EofException at org.neo4j.server.rest.repr.formats.StreamingJsonFormat$StreamingListWriter.writeValue(StreamingJsonFormat.java:316) ~[neo4j-server-2.1.0-M01.jar:2.1.0-M01] at org.neo4j.server.rest.repr.ListWriter.writeValue(ListWriter.java:75) ~[server-api-2.1.0-M01.jar:2.1.0-M01] at org.neo4j.server.rest.repr.ValueRepresentation.addTo(ValueRepresentation.java:49) ~[server-api-2.1.0-M01.jar:2.1.0-M01] at org.neo4j.server.rest.repr.ListRepresentation.serialize(ListRepresentation.java:65) ~[server-api-2.1.0-M01.jar:2.1.0-M01] at org.neo4j.server.rest.repr.Serializer.serialize(Serializer.java:75) ~[server-api-2.1.0-M01.jar:2.1.0-M01] at org.neo4j.server.rest.repr.MappingSerializer.putList(MappingSerializer.java:61) ~[server-api-2.1.0-M01.jar:2.1.0-M01] at org.neo4j.server.webadmin.rest.representations.RrdDataRepresentation$1.serialize(RrdDataRepresentation.java:68) ~[neo4j-server-2.1.0-M01.jar:2.1.0-M01] at org.neo4j.server.rest.repr.Serializer.serialize(Serializer.java:40) ~[server-api-2.1.0-M01.jar:2.1.0-M01] at org.neo4j.server.rest.repr.MappingSerializer.putMapping(MappingSerializer.java:56) ~[server-api-2.1.0-M01.jar:2.1.0-M01] at org.neo4j.server.rest.repr.MappingRepresentation.putTo(MappingRepresentation.java:57) ~[server-api-2.1.0-M01.jar:2.1.0-M01] at org.neo4j.server.rest.repr.ObjectRepresentation$PropertyGetter.putTo(ObjectRepresentation.java:133) ~[neo4j-server-2.1.0-M01.jar:2.1.0-M01] at org.neo4j.server.rest.repr.ObjectRepresentation.serialize(ObjectRepresentation.java:144) ~[neo4j-server-2.1.0-M01.jar:2.1.0-M01] at org.neo4j.server.rest.repr.MappingRepresentation.serialize(MappingRepresentation.java:41) ~[server-api-2.1.0-M01.jar:2.1.0-M01] at org.neo4j.server.rest.repr.OutputFormat$1.write(OutputFormat.java:164) ~[neo4j-server-2.1.0-M01.jar:2.1.0-M01] ... 30 common frames omitted Caused by: org.eclipse.jetty.io.EofException: null at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:186) ~[jetty-io-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.io.WriteFlusher.completeWrite(WriteFlusher.java:400) ~[jetty-io-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.io.SelectChannelEndPoint.onSelected(SelectChannelEndPoint.java:111) ~[jetty-io-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.io.SelectorManager$ManagedSelector.processKey(SelectorManager.java:498) ~[jetty-io-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:455) ~[jetty-io-9.0.5.v20130815.jar:9.0.5.v20130815] at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:420) ~[jetty-io-9.0.5.v20130815.jar:9.0.5.v20130815] ... 3 common frames omitted Caused by: java.io.IOException: Обрыв канала at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.7.0_51] at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.7.0_51] at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.7.0_51] at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.7.0_51] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) ~[na:1.7.0_51] at org.eclipse.jetty.io.ChannelEndPoint.flush(ChannelEndPoint.java:165) ~[jetty-io-9.0.5.v20130815.jar:9.0.5.v20130815] ... 8 common frames omitted

Upvotes: 1

Views: 288

Answers (2)

Michael Hunger
Michael Hunger

Reputation: 41686

So you have about 9M nodes in total, that means for MMIO you need about

  • 150MB for nodes

  • 500MB for rels (you can use 1G to be on the safe side)

    1. if your machine has only 8GB RAM you cannot assign 7GB heap b/c then you have no room left for memory mapping

    2. Use Periodic Commit for LOAD CSV, e.g. PERIODIC COMMIT 10000 LOAD CSV ...

Even loading your level 6 (8M nodes) with periodic commit should just take 5 mins.

  1. If you run your check script in parallel with your import, it will report what has been already committed, so this is expected behavior.

Did you run your check script on the master or the slaves? Do you have any push factors configured? Which would slow down the import massivly?

Perhaps import the data on a standalone db and then upload it to the master to use and sync?

Your query is quite expensive, in your full tree it would compute 8M paths, and compute all the payload of all the paths and then order the 8M rows and return them.

Nothing I'd do in the browser because of the timeout. I would probably use curl or a client with the transactional endpoint to stream the results and do the data extraction on the client side (if you need to).

You can also simplify the script to:

MATCH path = (root)-[r*]->(b)
WHERE id(root) = 0
RETURN length(path) as len, extract( n IN NODES(path) | n.payload ) AS payload_chain
ORDER BY len DESC

or better like you did:

MATCH path = (root:MyLabel {payload: 'root_payload'})-[r*]->(b)
RETURN length(path) as len, extract( n IN NODES(path) | n.payload ) AS payload_chain
ORDER BY len DESC

Upvotes: 2

Jacob Davis-Hansson
Jacob Davis-Hansson

Reputation: 2663

This is a known issue in M01, it will be resolved in the upcoming M02 release.

In general - we highly encourage experimenting with the Milestones, but especially low-numbered milestones often contain large changes (this one makes a big modification to the storage format, which is where the bug comes from), so make sure you don't depend on milestones for "real" projects.

Upvotes: 2

Related Questions