Cannot create cluster on ScyllaDB

Question

I just got started with ScyllaDB (on Windows 10, using Docker). Everything is fine when I create a node (docker run --name Node_X -d scylladb/scylla:4.1.0)

But when I create another one that I link to the 1st one (docker run --name Node_Y -d scylladb/scylla:4.1.0 --seeds="$(docker inspect --format='{{ .NetworkSettings.IPAddress }}' Node_X)").

It returns an error message: "nodetool: Unable to connect to Scylla API server: java.net.ConnectException: Connection refused (Connection refused) See 'nodetool help' or 'nodetool help '".

Many thanks for your help.

Guillaume

One can find the logs for the 2nd node below:

running: (['/opt/scylladb/scripts/scylla_dev_mode_setup', '--developer-mode', '1'],) running: (['/opt/scylladb/scripts/scylla_io_setup'],) 2020-12-28 14:07:02,470 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message. 2020-12-28 14:07:02,470 INFO Included extra file "/etc/supervisord.conf.d/node-exporter.conf" during parsing 2020-12-28 14:07:02,471 INFO Included extra file "/etc/supervisord.conf.d/rsyslog.conf" during parsing 2020-12-28 14:07:02,471 INFO Included extra file "/etc/supervisord.conf.d/scylla-housekeeping.conf" during parsing 2020-12-28 14:07:02,471 INFO Included extra file "/etc/supervisord.conf.d/scylla-jmx.conf" during parsing 2020-12-28 14:07:02,471 INFO Included extra file "/etc/supervisord.conf.d/scylla-server.conf" during parsing 2020-12-28 14:07:02,471 INFO Included extra file "/etc/supervisord.conf.d/sshd-server.conf" during parsing 2020-12-28 14:07:02,504 INFO RPC interface 'supervisor' initialized 2020-12-28 14:07:02,504 CRIT Server 'inet_http_server' running without any HTTP authentication checking 2020-12-28 14:07:02,504 INFO supervisord started with pid 26 2020-12-28 14:07:03,507 INFO spawned: 'scylla' with pid 29 2020-12-28 14:07:03,509 INFO spawned: 'scylla-housekeeping' with pid 30 2020-12-28 14:07:03,511 INFO spawned: 'sshd' with pid 31 2020-12-28 14:07:03,512 INFO spawned: 'scylla-jmx' with pid 32 2020-12-28 14:07:03,514 INFO spawned: 'rsyslog' with pid 33 2020-12-28 14:07:03,515 INFO spawned: 'node-exporter' with pid 34 Generating public/private ed25519 key pair. Your identification has been saved in /etc/ssh/ssh_host_ed25519_key. Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub. The key fingerprint is: SHA256:h6ScBFQ3ZuJpK3F0GZUTfOmLinXX1+LxW7B8rxOx5FM root@f881e3d82b89 The key's randomart image is: +--[ED25519 256]--+ | .o.+ B=oo . | | + B..+ o | | . * . + | | * = . . o E| | . = S .. =.+.| | . ..o o.Ooo| | o o . .oBo| | . . o.+| | .+o| +----[SHA256]-----+ time="2020-12-28T14:07:03Z" level=info msg="Starting node_exporter (version=0.17.0, branch=HEAD, revision=f6f6194a436b9a63d0439abc585c76b19a206b21)" source="node_exporter.go:82" time="2020-12-28T14:07:03Z" level=info msg="Build context (go=go1.11.2, user=root@322511e06ced, date=20181130-15:51:33)" source="node_exporter.go:83" time="2020-12-28T14:07:03Z" level=info msg="Enabled collectors:" source="node_exporter.go:90" time="2020-12-28T14:07:03Z" level=info msg=" - arp" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - bcache" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - bonding" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - conntrack" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - cpu" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - diskstats" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - edac" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - entropy" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - filefd" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - filesystem" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - hwmon" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - infiniband" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - interrupts" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - ipvs" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - loadavg" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - mdadm" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - meminfo" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - netclass" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - netdev" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - netstat" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - nfs" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - nfsd" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - sockstat" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - stat" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - textfile" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - time" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - timex" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - uname" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - vmstat" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - xfs" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg=" - zfs" source="node_exporter.go:97" time="2020-12-28T14:07:03Z" level=info msg="Listening on :9100" source="node_exporter.go:111" /etc/scylla.d/docker.conf: line 1: docker: command not found FATAL: Exception during startup, aborting: boost::wrapexceptboost::bad_lexical_cast (bad lexical cast: source type value could not be interpreted as target) 2020-12-28 14:07:03,984 INFO exited: scylla (exit status 7; not expected) Connecting to http://localhost:10000 Starting the JMX server Generating public/private rsa key pair. Your identification has been saved in /etc/ssh/ssh_host_rsa_key. Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub. The key fingerprint is: SHA256:q+U05wLwl2LeoE98MrGp7MHuEpk4021nzP+TKeIVZbM root@f881e3d82b89 The key's randomart image is: +---[RSA 4096]----+ | | | | | + | | . o o | | o + =. S.E | |+ =.o.@+oo | | o oo*BBB .o | | .o.+oX=++ | | +*o+..+o. | +----[SHA256]-----+ Could not load host key: /etc/ssh/ssh_host_ecdsa_key JMX is enabled to receive remote connections on port: 7199 2020-12-28 14:07:05,423 INFO spawned: 'scylla' with pid 73 2020-12-28 14:07:05,424 INFO success: scylla-housekeeping entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2020-12-28 14:07:05,424 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2020-12-28 14:07:05,424 INFO success: scylla-jmx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2020-12-28 14:07:05,424 INFO success: rsyslog entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2020-12-28 14:07:05,424 INFO success: node-exporter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) /etc/scylla.d/docker.conf: line 1: docker: command not found FATAL: Exception during startup, aborting: boost::wrapexceptboost::bad_lexical_cast (bad lexical cast: source type value could not be interpreted as target) 2020-12-28 14:07:05,979 INFO exited: scylla (exit status 7; not expected) 2020-12-28 14:07:07,986 INFO spawned: 'scylla' with pid 181 /etc/scylla.d/docker.conf: line 1: docker: command not found FATAL: Exception during startup, aborting: boost::wrapexceptboost::bad_lexical_cast (bad lexical cast: source type value could not be interpreted as target) 2020-12-28 14:07:08,525 INFO exited: scylla (exit status 7; not expected) multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 1319, in do_open encode_chunked=req.has_header('Transfer-encoding')) File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 1252, in request self._send_request(method, url, body, headers, encode_chunked) File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 1298, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 1247, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 1026, in _send_output self.send(msg) File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 966, in send self.connect() File "/opt/scylladb/python3/lib64/python3.7/http/client.py", line 938, in connect (self.host,self.port), self.timeout, self.source_address) File "/opt/scylladb/python3/lib64/python3.7/socket.py", line 728, in create_connection raise err File "/opt/scylladb/python3/lib64/python3.7/socket.py", line 716, in create_connection sock.connect(sa) OSError: [Errno 99] Cannot assign requested address

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/scylladb/python3/lib64/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, **kwds)) File "/opt/scylladb/scripts/libexec/scylla-housekeeping", line 64, in get_url return urllib.request.urlopen(path).read().decode('utf-8') File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 222, in urlopen return opener.open(url, data, timeout) File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 525, in open response = self._open(req, data) File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 543, in _open '_open', req) File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 503, in _call_chain result = func(*args) File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 1347, in http_open return self.do_open(http.client.HTTPConnection, req) File "/opt/scylladb/python3/lib64/python3.7/urllib/request.py", line 1321, in do_open raise URLError(err) urllib.error.URLError: <urlopen error [Errno 99] Cannot assign requested address> """

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/opt/scylladb/scripts/libexec/scylla-housekeeping", line 197, in args.func(args) File "/opt/scylladb/scripts/libexec/scylla-housekeeping", line 123, in check_version current_version = sanitize_version(get_api('/storage_service/scylla_release_version'))
File "/opt/scylladb/scripts/libexec/scylla-housekeeping", line 82, in get_api return get_json_from_url("http://" + api_address + path) File "/opt/scylladb/scripts/libexec/scylla-housekeeping", line 74, in get_json_from_url retval = result.get(timeout=5) File "/opt/scylladb/python3/lib64/python3.7/multiprocessing/pool.py", line 657, in get raise self._value urllib.error.URLError: <urlopen error [Errno 99] Cannot assign requested address> 2020-12-28 14:07:12,202 INFO spawned: 'scylla' with pid 216 /etc/scylla.d/docker.conf: line 1: docker: command not found FATAL: Exception during startup, aborting: boost::wrapexceptboost::bad_lexical_cast (bad lexical cast: source type value could not be interpreted as target) 2020-12-28 14:07:12,727 INFO exited: scylla (exit status 7; not expected) 2020-12-28 14:07:13,729 INFO gave up: scylla entered FATAL state, too many start retries too quickly

Answer 1

Your port forwarding is likely to be wrong, the two instances need to connect, how's your docker invocation line looks like?

Answer 2

I shared your post with ScyllaDB engineers. Hopefully they will respond here. If you want, you can also post your question in our community slack here:

http://slack.scylladb.com/

Or our user group:

https://groups.google.com/g/scylladb-users