CODEWITHSUNDEEP
neo4jconfiggraphaware
Zhenshan Jin
Zhenshan Jin

Reputation: 419

Elasticsearch can't sync 2GB database from neo4j with GprahAware

The neo4j2elasticsearch works on my machine when the database is only 250KB. But the the databse is around 2GB. It won't sync anymore. I'm wondering is because of these parameters in the config file:

#optional, size of the in-memory queue that queues up operations to be synchronised to Elasticsearch, defaults to 10000
com.graphaware.module.ES.queueSize=10000

#optional, size of the batch size to use during re-initialization, defaults to 1000
com.graphaware.module.ES.reindexBatchSize=2000

I'm wondering what is the unit of in-memory queue size 10000 and how to estimate what parameter to set based on my database size.

Here is the debug file : neo4j debug.log failure loading

The database is re-initialized but there are only empty neo4j-index-relationship/neo4j-index-node index in the elasticsearch database

Just for information, here is the debug file for successful 250KB database loading: neo4j debug.log successful loading seems like Re-indexing nodes... step is missing in the 2GB database loading procedure.

Upvotes: 1

Views: 114

Answers (1)

Isaac Rosado
Isaac Rosado

Reputation: 1029

The log doesn't have a line saying it will re-index, did you configure:

com.graphaware.module.ES.initializeUntil=

To a timestamp that warrants re-indexing on startup? Otherwise, it will only index new data. It is explained at the bottom of https://github.com/graphaware/neo4j-to-elasticsearch :

...in order to trigger (re-)indexing, i.e. sending every node that should be indexed to Elasticsearch upon Neo4j restart, you have to manually intervene...

So try to create a new node and see if the synchronization is working for new stuff to eliminate this situation (most common one).

Upvotes: 0

Related Questions