CODEWITHSUNDEEP
mysqlnosqlneo4jgraph-databases
hablema
hablema

Reputation: 550

how to speed up insertion in neo4j from mysql?

I have a dataset of 60000 items in mysql and i'm trying to insert it into neo4j. The insertion is taking place but it's taking a long time( approx. 10-15 per 3 sec). Is there any way i can speed it up? also is there any way i could give something such as unique key in neo4j so duplication indexes won't get indexed? I'm new to neo4j.

I'm using neo4j 1.8 with PHP Everyman driver.

Upvotes: 2

Views: 1800

Answers (2)

Roelb
Roelb

Reputation: 46

As mentioned above, the preferred option is the batch importer.

If you need to go through the PHP client, here I've put up an example that uses the REST batch api: http://phpfiddle.org/main/code/mu3-sgk

You can test the rows/batch what works best for your system. For my laptop it's 750, for my test-server it's 1250. The json_decode that happens is heavy on the CPU.

Upvotes: 1

Michael Hunger
Michael Hunger

Reputation: 41706

There is a nice presentation from Max De Marzi, about ETL into Neo4j.

See: http://www.slideshare.net/maxdemarzi/etl-into-neo4j

It depends which language you want to use, lots of options from java embedded via jruby and remotely via ruby, php, python.

You would want to batch your requests in appropriately sized transactions (e.g. 10k items per tx).

It is possible to import CSV files directly into a database file using my the batch-importer or via the BATCH REST API of the Neo4j Server.

Upvotes: 2

Related Questions