Stand Up Neo4j HA on AWS Elastic Beanstalk

Question

I have a Java web application that uses Neo4J embedded that I'm trying to migrate to Neo4j HA mode.

I'm hosting in Amazon Elastic Beanstalk.

Part of my cluster architecture I created a dedicated machine in my Virtual Private Cloud that I want to be the "Master" and for all EC2 instance stood up in the Beanstalk application to connect to it.

So my 2 part question:

Does each member of the cluster need to know of the existence of every EC2 instance that is stood up in the application? I'll need a way of registering a new server with the cluster whenever Elastic Beanstalk auto scales?

I'm just trying to get Neo4J stood up on the master server and I get this error:

Starting Neo4j Server failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@1b7440c3' was successfully initialized, but failed to start. Please see attached cause exception.
    at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:258) ~[neo4j-server-2.2.5.jar:2.2.5]
    at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:117) [neo4j-server-2.2.5.jar:2.2.5]
    at org.neo4j.server.Bootstrapper.main(Bootstrapper.java:69) [neo4j-server-2.2.5.jar:2.2.5]
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.database.LifecycleManagingDatabase@1b7440c3' was successfully initialized, but failed to start. Please see attached cause exception.
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:513) ~[neo4j-kernel-2.2.5.jar:2.2.5]
    at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:115) ~[neo4j-kernel-2.2.5.jar:2.2.5]
    at org.neo4j.server.AbstractNeoServer.start(AbstractNeoServer.java:203) ~[neo4j-server-2.2.5.jar:2.2.5]
    ... 2 common frames omitted
Caused by: java.lang.RuntimeException: Error starting org.neo4j.kernel.ha.HighlyAvailableGraphDatabase, /usr/local/neo4j/data/graph.db
    at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:335) ~[neo4j-kernel-2.2.5.jar:2.2.5]
    at org.neo4j.kernel.ha.HighlyAvailableGraphDatabase.(HighlyAvailableGraphDatabase.java:173) ~[neo4j-ha-2.2.5.jar:2.2.5]
    at org.neo4j.server.enterprise.EnterpriseNeoServer$1.newGraphDatabase(EnterpriseNeoServer.java:61) ~[neo4j-server-enterprise-2.2.5.jar:2.2.5]
    at org.neo4j.server.database.LifecycleManagingDatabase.start(LifecycleManagingDatabase.java:110) ~[neo4j-server-2.2.5.jar:2.2.5]
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:507) ~[neo4j-kernel-2.2.5.jar:2.2.5]
    ... 4 common frames omitted
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.lifecycle.LifeSupport@cdbac7f' was successfully initialized, but failed to start. Please see attached cause exception.
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:513) ~[neo4j-kernel-2.2.5.jar:2.2.5]
    at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:115) ~[neo4j-kernel-2.2.5.jar:2.2.5]
    at org.neo4j.kernel.InternalAbstractGraphDatabase.run(InternalAbstractGraphDatabase.java:331) ~[neo4j-kernel-2.2.5.jar:2.2.5]
    ... 8 common frames omitted
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.cluster.client.ClusterClient@f2c6d70' was successfully initialized, but failed to start. Please see attached cause exception.
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:513) ~[neo4j-kernel-2.2.5.jar:2.2.5]
    at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:115) ~[neo4j-kernel-2.2.5.jar:2.2.5]
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:507) ~[neo4j-kernel-2.2.5.jar:2.2.5]
    ... 10 common frames omitted
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.cluster.client.ClusterJoin@342a3fad' was successfully initialized, but failed to start. Please see attached cause exception.
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:513) ~[neo4j-kernel-2.2.5.jar:2.2.5]
    at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:115) ~[neo4j-kernel-2.2.5.jar:2.2.5]
    at org.neo4j.cluster.client.ClusterClient.start(ClusterClient.java:423) ~[neo4j-cluster-2.2.5.jar:2.2.5]
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:507) ~[neo4j-kernel-2.2.5.jar:2.2.5]
    ... 12 common frames omitted
Caused by: java.lang.RuntimeException: java.net.UnknownHostException: ip-10-0-0-185: ip-10-0-0-185: unknown error
    at org.neo4j.cluster.client.ClusterJoin.resolvePortOnlyHost(ClusterJoin.java:228) ~[neo4j-cluster-2.2.5.jar:2.2.5]
    at org.neo4j.cluster.client.ClusterJoin.access$200(ClusterJoin.java:52) ~[neo4j-cluster-2.2.5.jar:2.2.5]
    at org.neo4j.cluster.client.ClusterJoin$2.apply(ClusterJoin.java:144) ~[neo4j-cluster-2.2.5.jar:2.2.5]
    at org.neo4j.cluster.client.ClusterJoin$2.apply(ClusterJoin.java:140) ~[neo4j-cluster-2.2.5.jar:2.2.5]
    at org.neo4j.helpers.collection.Iterables$MapIterable$MapIterator.next(Iterables.java:670) ~[neo4j-kernel-2.2.5.jar:2.2.5]
    at org.neo4j.helpers.collection.Iterables.addAll(Iterables.java:188) ~[neo4j-kernel-2.2.5.jar:2.2.5]
    at org.neo4j.helpers.collection.Iterables.toList(Iterables.java:555) ~[neo4j-kernel-2.2.5.jar:2.2.5]
    at org.neo4j.helpers.collection.Iterables.toArray(Iterables.java:581) ~[neo4j-kernel-2.2.5.jar:2.2.5]
    at org.neo4j.cluster.client.ClusterJoin.joinByConfig(ClusterJoin.java:138) ~[neo4j-cluster-2.2.5.jar:2.2.5]
    at org.neo4j.cluster.client.ClusterJoin.start(ClusterJoin.java:91) ~[neo4j-cluster-2.2.5.jar:2.2.5]
    at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:507) ~[neo4j-kernel-2.2.5.jar:2.2.5]
    ... 15 common frames omitted
Caused by: java.net.UnknownHostException: ip-10-0-0-185: ip-10-0-0-185: unknown error
    at java.net.InetAddress.getLocalHost(InetAddress.java:1505) ~[na:1.8.0_51]
    at org.neo4j.cluster.client.ClusterJoin.resolvePortOnlyHost(ClusterJoin.java:224) ~[neo4j-cluster-2.2.5.jar:2.2.5]
    ... 25 common frames omitted
Caused by: java.net.UnknownHostException: ip-10-0-0-185: unknown error
    at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) ~[na:1.8.0_51]
    at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) ~[na:1.8.0_51]
    at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) ~[na:1.8.0_51]
    at java.net.InetAddress.getLocalHost(InetAddress.java:1500) ~[na:1.8.0_51]
    ... 26 common frames omitted

neo4j.properties and neo4j-server.properties included

Neo4j.properties:

Enable online backups to be taken from this database. online_backup_enabled=true



# Port to listen to for incoming backup requests.
online_backup_server=127.0.0.1:6362


# Uncomment and specify these lines for running Neo4j in High Availability mode.
# See the High availability setup tutorial for more details on these settings
# http://neo4j.com/docs/2.2.5/ha-setup-tutorial.html

# ha.server_id is the number of each instance in the HA cluster. It should be
# an integer (e.g. 1), and should be unique for each cluster instance.
ha.server_id=1

# ha.initial_hosts is a comma-separated list (without spaces) of the host:port
# where the ha.cluster_server of all instances will be listening. Typically
# this will be the same for all cluster instances.
ha.initial_hosts=neo4jserver:5001

# IP and port for this instance to listen on, for communicating cluster status
# information iwth other instances (also see ha.initial_hosts). The IP
# must be the configured IP address for one of the local interfaces.
ha.cluster_server=neo4jserver:5001

# IP and port for this instance to listen on, for communicating transaction
# data with other instances (also see ha.initial_hosts). The IP
# must be the configured IP address for one of the local interfaces.
ha.server=neo4jserver:6001

# The interval at which slaves will pull updates from the master. Comment out
# the option to disable periodic pulling of updates. Unit is seconds.
ha.pull_interval=10

# Amount of slaves the master will try to push a transaction to upon commit
# (default is 1). The master will optimistically continue and not fail the
# transaction even if it fails to reach the push factor. Setting this to 0 will
# increase write performance when writing through master but could potentially
# lead to branched data (or loss of transaction) if the master goes down.
#ha.tx_push_factor=1

Neo4j-server.properties:



# location of the database directory
org.neo4j.server.database.location=data/graph.db

# Low-level graph engine tuning file
org.neo4j.server.db.tuning.properties=conf/neo4j.properties

# Database mode
# Allowed values:
# HA - High Availability
# SINGLE - Single mode, default.
# To run in High Availability mode, configure the neo4j.properties config file, then uncomment this line:
org.neo4j.server.database.mode=HA

# Let the webserver only listen on the specified IP. Default is localhost (only
# accept local connections). Uncomment to allow any connection. Please see the
# security section in the neo4j manual before modifying this.
org.neo4j.server.webserver.address=0.0.0.0

# Require (or disable the requirement of) auth to access Neo4j
dbms.security.auth_enabled=true

#
# HTTP Connector
#

# http port (for all data, administrative, and UI access)
org.neo4j.server.webserver.port=7474

#
# HTTPS Connector
#

# Turn https-support on/off
org.neo4j.server.webserver.https.enabled=true

# https port (for all data, administrative, and UI access)
org.neo4j.server.webserver.https.port=7473

# Certificate location (auto generated if the file does not exist)
org.neo4j.server.webserver.https.cert.location=conf/ssl/snakeoil.cert

# Private key location (auto generated if the file does not exist)
org.neo4j.server.webserver.https.key.location=conf/ssl/snakeoil.key

# Internally generated keystore (don't try to put your own
# keystore there, it will get deleted when the server starts)
org.neo4j.server.webserver.https.keystore.location=data/keystore

# Comma separated list of JAX-RS packages containing JAX-RS resources, one
# package name for each mountpoint. The listed package names will be loaded
# under the mountpoints specified. Uncomment this line to mount the
# org.neo4j.examples.server.unmanaged.HelloWorldResource.java from
# neo4j-server-examples under /examples/unmanaged, resulting in a final URL of
# http://localhost:7474/examples/unmanaged/helloworld/{nodeId}
#org.neo4j.server.thirdparty_jaxrs_classes=org.neo4j.examples.server.unmanaged=/examples/unmanaged


#*****************************************************************
# HTTP logging configuration
#*****************************************************************

# HTTP logging is disabled. HTTP logging can be enabled by setting this
# property to 'true'.
org.neo4j.server.http.log.enabled=false

# Logging policy file that governs how HTTP log output is presented and
# archived. Note: changing the rollover and retention policy is sensible, but
# changing the output format is less so, since it is configured to use the
# ubiquitous common log format
org.neo4j.server.http.log.config=conf/neo4j-http-logging.xml

#*****************************************************************
# Administration client configuration
#*****************************************************************

# location of the servers round-robin database directory. possible values:
# - absolute path like /var/rrd
# - path relative to the server working directory like data/rrd
# - commented out, will default to the database data directory.
org.neo4j.server.webadmin.rrdb.location=data/rrd

MicTech · Accepted Answer

When cluster starting, then Neo4j using ha.initial_hosts property for initialize cluster on the beginning. When your cluster is up and you trying to add new machine into the cluster it's enough to have a knowledge about the cluster on the new machine. It will attach to the cluster.
It looks like a problem with connection between machines or translation from hostname to ip address.

Please provide your neo4j-server.properties and log messages.

Stand Up Neo4j HA on AWS Elastic Beanstalk

Answers (1)

Related Questions