CODEWITHSUNDEEP
dockercassandrascylla
sakura-bloom
sakura-bloom

Reputation: 4594

Cannot run nodetool commands and cqlsh to Scylla in Docker

I am new to Scylla and I am following the instructions to try it in a container as per this page: https://hub.docker.com/r/scylladb/scylla/.

The following command ran fine.

docker run --name some-scylla --hostname some-scylla -d scylladb/scylla

I see the container is running.

CONTAINER ID   IMAGE             COMMAND                  CREATED          STATUS          PORTS                                                            NAMES
e6c4e19ff1bd   scylladb/scylla   "/docker-entrypoint.…"   14 seconds ago   Up 13 seconds   22/tcp, 7000-7001/tcp, 9042/tcp, 9160/tcp, 9180/tcp, 10000/tcp   some-scylla

However, I'm unable to use nodetool or cqlsh. I get the following output.

$ docker exec -it some-scylla nodetool status
Using /etc/scylla/scylla.yaml as the config file
nodetool: Unable to connect to Scylla API server: java.net.ConnectException: Connection refused (Connection refused)
See 'nodetool help' or 'nodetool help <command>'.

and

$ docker exec -it some-scylla cqlsh
Connection error: ('Unable to connect to any servers', {'172.17.0.2': error(111, "Tried connecting to [('172.17.0.2', 9042)]. Last error: Connection refused")})

Any ideas?

Update

Looking at docker logs some-scylla I see some errors in the logs, the last one is as follows.

2021-10-03 07:51:04,771 INFO spawned: 'scylla' with pid 167
Scylla version 4.4.4-0.20210801.69daa9fd0 with build-id eb11cddd30e88ef39c32c847e70181b5cf786355 starting ...
command used: "/usr/bin/scylla --log-to-syslog 0 --log-to-stdout 1 --default-log-level info --network-stack posix --developer-mode=1 --overprovisioned --listen-address 172.17.0.2 --rpc-address 172.17.0.2 --seed-provider-parameters seeds=172.17.0.2 --blocked-reactor-notify-ms 999999999"
parsed command line options: [log-to-syslog: 0, log-to-stdout: 1, default-log-level: info, network-stack: posix, developer-mode: 1, overprovisioned, listen-address: 172.17.0.2, rpc-address: 172.17.0.2, seed-provider-parameters: seeds=172.17.0.2, blocked-reactor-notify-ms: 999999999]
ERROR 2021-10-03 07:51:05,203 [shard 6] seastar - Could not setup Async I/O: Resource temporarily unavailable. The most common cause is not enough request capacity in /proc/sys/fs/aio-max-nr. Try increasing that number or reducing the amount of logical CPUs available for your application
2021-10-03 07:51:05,316 INFO exited: scylla (exit status 1; not expected)
2021-10-03 07:51:06,318 INFO gave up: scylla entered FATAL state, too many start retries too quickly

Update 2

The reason for the error was described on the docker hub page linked above. I had to start container specifying the number of CPUs with --smp 1 as follows.

docker run --name some-scylla --hostname some-scylla -d scylladb/scylla --smp 1

According to the above page:

This command will start a Scylla single-node cluster in developer mode (see --developer-mode 1) limited by a single CPU core (see --smp). Production grade configuration requires tuning a few kernel parameters such that limiting number of available cores (with --smp 1) is the simplest way to go.

Multiple cores requires setting a proper value to the /proc/sys/fs/aio-max-nr. On many non production systems it will be equal to 65K. ...

Upvotes: 5

Views: 1957

Answers (1)

Felipe Mendes
Felipe Mendes

Reputation: 281

As you have found out, in order to be able to use additional CPU cores you'll need to increase fs.aio-max-nr kernel parameter.

You may run as root:

# sysctl -w fs.aio-max-nr=65535

Which should be enough for most systems. Should you still have any error preventing it to use all of your CPU cores, increase its value further.

Do notice that the above configuration is not persistent. Edit /etc/sysctl.conf in order to make it persistent across reboots.

Upvotes: 6

Related Questions