Riak erlang error on startup

Question

I run riak as such

[idf@node1 ~]$ sudo riak start
!!!!
!!!! WARNING: ulimit -n is 65535; 65536 is the recommended minimum.
!!!!
riak failed to start within 15 seconds,
see the output of 'riak console' for more information.
If you want to wait longer, set the environment variable
WAIT_FOR_ERLANG to the number of seconds to wait.
[idf@node1 ~]$ 

It tells me to look at the output of riak console

[idf@node1 ~]$ sudo riak console
config is OK
-config /var/lib/riak/generated.configs/app.2016.04.04.21.34.43.config -args_file /var/lib/riak/generated.configs/vm.2016.04.04.21.34.43.args -vm_args /var/lib/riak/generated.configs/vm.2016.04.04.21.34.43.args
!!!!
!!!! WARNING: ulimit -n is 65535; 65536 is the recommended minimum.
!!!!
Exec:  /usr/lib64/riak/erts-5.10.3/bin/erlexec -boot /usr/lib64/riak/releases/2.0.0/riak               -config /var/lib/riak/generated.configs/app.2016.04.04.21.34.43.config -args_file /var/lib/riak/generated.configs/vm.2016.04.04.21.34.43.args -vm_args /var/lib/riak/generated.configs/vm.2016.04.04.21.34.43.args              -pa /usr/lib64/riak/lib/basho-patches -- console
Root: /usr/lib64/riak
Erlang R16B02-basho5 (erts-5.10.3) [source] [64-bit] [smp:16:16] [async-threads:64] [kernel-poll:true]

[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
{"Kernel pid terminated",application_controller,"{application_start_failure,riak_core,{{shutdown,{failed_to_start_child,riak_core_handoff_sup,{shutdown,{failed_to_start_child,riak_core_handoff_listener_sup,{shutdown,{failed_to_start_child,riak_core_handoff_listener,{bad_return_value,{error,eaddrinuse}}}}}}}},{riak_core_app,start,[normal,[]]}}}"}

Crash dump was written to: /var/log/riak/erl_crash.dump
Kernel pid terminated (application_controller) ({application_start_failure,riak_core,{{shutdown,{failed_to_start_child,riak_core_handoff_sup,{shutdown,{failed_to_start_child,riak_core_handoff_listen
[idf@node1 ~]$ 

EDIT:

I changed the default values in the .conf file to the following:

listener.http.internal = 127.0.0.1:10011
listener.protobuf.internal = 127.0.0.1:10012

I still get:

[idf@node1 ~]$ sudo riak console
config is OK
-config /var/lib/riak/generated.configs/app.2016.04.05.11.55.10.config -args_file /var/lib/riak/generated.configs/vm.2016.04.05.11.55.10.args -vm_args /var/lib/riak/generated.configs/vm.2016.04.05.11.55.10.args
!!!!
!!!! WARNING: ulimit -n is 65535; 65536 is the recommended minimum.
!!!!
Exec:  /usr/lib64/riak/erts-5.10.3/bin/erlexec -boot /usr/lib64/riak/releases/2.0.0/riak               -config /var/lib/riak/generated.configs/app.2016.04.05.11.55.10.config -args_file /var/lib/riak/generated.configs/vm.2016.04.05.11.55.10.args -vm_args /var/lib/riak/generated.configs/vm.2016.04.05.11.55.10.args              -pa /usr/lib64/riak/lib/basho-patches -- console
Root: /usr/lib64/riak
Erlang R16B02-basho5 (erts-5.10.3) [source] [64-bit] [smp:16:16] [async-threads:64] [kernel-poll:true]

[os_mon] memory supervisor port (memsup): Erlang has closed
[os_mon] cpu supervisor port (cpu_sup): Erlang has closed
{"Kernel pid terminated",application_controller,"{application_start_failure,riak_core,{{shutdown,{failed_to_start_child,riak_core_handoff_sup,{shutdown,{failed_to_start_child,riak_core_handoff_listener_sup,{shutdown,{failed_to_start_child,riak_core_handoff_listener,{bad_return_value,{error,eaddrinuse}}}}}}}},{riak_core_app,start,[normal,[]]}}}"}

Crash dump was written to: /var/log/riak/erl_crash.dump
Kernel pid terminated (application_controller) ({application_start_failure,riak_core,{{shutdown,{failed_to_start_child,riak_core_handoff_sup,{shutdown,{failed_to_start_child,riak_core_handoff_listen

EDIT 2

I see that even though it fails, there is a riak daemon running?

[idf@node1 ~]$ ps ax | grep riak
26817 ?        S      0:00 /usr/lib64/riak/erts-5.10.3/bin/epmd -daemon
27177 pts/3    S+     0:00 grep --color=auto riak

In the documentation, it says to look for beam.smp, but I don't see those.

"exit with reason bad return value: {error,eaddrinuse} in context start_error An error like this example can occur when another process is already bound to the same address as the process being started is attempting to bind to. Use operating system tools like netstat, ps, and lsof to determine the root cause for resolving this kind of errors; check for existence of stale beam.smp processes."

[idf@node1 ~]$ ps ax | grep -i beam
27217 pts/3    S+     0:00 grep --color=auto -i beam
[idf@node1 ~]$

This is the .conf file

## Where to emit the default log messages (typically at 'info'
## severity):
## off: disabled
## file: the file specified by log.console.file
## console: to standard output (seen when using `riak attach-direct`)
## both: log.console.file and standard out.
## 
## Default: file
## 
## Acceptable values:
##   - one of: off, file, console, both
log.console = file

## The severity level of the console log, default is 'info'.
## 
## Default: info
## 
## Acceptable values:
##   - one of: debug, info, notice, warning, error, critical, alert, emergency, none
log.console.level = info

## When 'log.console' is set to 'file' or 'both', the file where
## console messages will be logged.
## 
## Default: $(platform_log_dir)/console.log
## 
## Acceptable values:
##   - the path to a file
log.console.file = $(platform_log_dir)/console.log

## The file where error messages will be logged.
## 
## Default: $(platform_log_dir)/error.log
## 
## Acceptable values:
##   - the path to a file
log.error.file = $(platform_log_dir)/error.log

## When set to 'on', enables log output to syslog.
## 
## Default: off
## 
## Acceptable values:
##   - on or off
log.syslog = off

## Whether to enable the crash log.
## 
## Default: on
## 
## Acceptable values:
##   - on or off
log.crash = on

## If the crash log is enabled, the file where its messages will
## be written.
## 
## Default: $(platform_log_dir)/crash.log
## 
## Acceptable values:
##   - the path to a file
log.crash.file = $(platform_log_dir)/crash.log

## Maximum size in bytes of individual messages in the crash log
## 
## Default: 64KB
## 
## Acceptable values:
##   - a byte size with units, e.g. 10GB
log.crash.maximum_message_size = 64KB

## Maximum size of the crash log in bytes, before it is rotated
## 
## Default: 10MB
## 
## Acceptable values:
##   - a byte size with units, e.g. 10GB
log.crash.size = 10MB

## The schedule on which to rotate the crash log.  For more
## information see:
## https://github.com/basho/lager/blob/master/README.md#internal-log-rotation
## 
## Default: $D0
## 
## Acceptable values:
##   - text
log.crash.rotation = $D0

## The number of rotated crash logs to keep. When set to
## 'current', only the current open log file is kept.
## 
## Default: 5
## 
## Acceptable values:
##   - an integer
##   - the text "current"
log.crash.rotation.keep = 5

## Name of the Erlang node
## 
## Default: riak@127.0.0.1
## 
## Acceptable values:
##   - text
nodename = riak@127.0.0.1

## Cookie for distributed node communication.  All nodes in the
## same cluster should use the same cookie or they will not be able to
## communicate.
## 
## Default: riak
## 
## Acceptable values:
##   - text
distributed_cookie = riak

## Sets the number of threads in async thread pool, valid range
## is 0-1024. If thread support is available, the default is 64.
## More information at: http://erlang.org/doc/man/erl.html
## 
## Default: 64
## 
## Acceptable values:
##   - an integer
erlang.async_threads = 64

## The number of concurrent ports/sockets
## Valid range is 1024-134217727
## 
## Default: 65536
## 
## Acceptable values:
##   - an integer
erlang.max_ports = 65535

## Set scheduler forced wakeup interval. All run queues will be
## scanned each Interval milliseconds. While there are sleeping
## schedulers in the system, one scheduler will be woken for each
## non-empty run queue found. An Interval of zero disables this
## feature, which also is the default.
## This feature is a workaround for lengthy executing native code, and
## native code that do not bump reductions properly.
## More information: http://www.erlang.org/doc/man/erl.html#+sfwi
## 
## Acceptable values:
##   - an integer
## erlang.schedulers.force_wakeup_interval = 500

## Enable or disable scheduler compaction of load. By default
## scheduler compaction of load is enabled. When enabled, load
## balancing will strive for a load distribution which causes as many
## scheduler threads as possible to be fully loaded (i.e., not run out
## of work). This is accomplished by migrating load (e.g. runnable
## processes) into a smaller set of schedulers when schedulers
## frequently run out of work. When disabled, the frequency with which
## schedulers run out of work will not be taken into account by the
## load balancing logic.
## More information: http://www.erlang.org/doc/man/erl.html#+scl
## 
## Acceptable values:
##   - one of: true, false
## erlang.schedulers.compaction_of_load = false

## Enable or disable scheduler utilization balancing of load. By
## default scheduler utilization balancing is disabled and instead
## scheduler compaction of load is enabled which will strive for a
## load distribution which causes as many scheduler threads as
## possible to be fully loaded (i.e., not run out of work). When
## scheduler utilization balancing is enabled the system will instead
## try to balance scheduler utilization between schedulers. That is,
## strive for equal scheduler utilization on all schedulers.
## More information: http://www.erlang.org/doc/man/erl.html#+sub
## 
## Acceptable values:
##   - one of: true, false
## erlang.schedulers.utilization_balancing = true

## Number of partitions in the cluster (only valid when first
## creating the cluster). Must be a power of 2, minimum 8 and maximum
## 1024.
## 
## Default: 64
## 
## Acceptable values:
##   - an integer
## ring_size = 64

## Number of concurrent node-to-node transfers allowed.
## 
## Default: 2
## 
## Acceptable values:
##   - an integer
## transfer_limit = 2

## Default cert location for https can be overridden
## with the ssl config variable, for example:
## 
## Acceptable values:
##   - the path to a file
## ssl.certfile = $(platform_etc_dir)/cert.pem

## Default key location for https can be overridden with the ssl
## config variable, for example:
## 
## Acceptable values:
##   - the path to a file
## ssl.keyfile = $(platform_etc_dir)/key.pem

## Default signing authority location for https can be overridden
## with the ssl config variable, for example:
## 
## Acceptable values:
##   - the path to a file
## ssl.cacertfile = $(platform_etc_dir)/cacertfile.pem

## DTrace support Do not enable 'dtrace' unless your Erlang/OTP
## runtime is compiled to support DTrace.  DTrace is available in
## R15B01 (supported by the Erlang/OTP official source package) and in
## R14B04 via a custom source repository & branch.
## 
## Default: off
## 
## Acceptable values:
##   - on or off
dtrace = off

## Platform-specific installation paths (substituted by rebar)
## 
## Default: /usr/sbin
## 
## Acceptable values:
##   - the path to a directory
platform_bin_dir = /usr/sbin

## 
## Default: /var/lib/riak
## 
## Acceptable values:
##   - the path to a directory
platform_data_dir = /var/lib/riak

## 
## Default: /etc/riak
## 
## Acceptable values:
##   - the path to a directory
platform_etc_dir = /etc/riak

## 
## Default: /usr/lib64/riak/lib
## 
## Acceptable values:
##   - the path to a directory
platform_lib_dir = /usr/lib64/riak/lib

## 
## Default: /var/log/riak
## 
## Acceptable values:
##   - the path to a directory
platform_log_dir = /var/log/riak

## Enable consensus subsystem. Set to 'on' to enable the
## consensus subsystem used for strongly consistent Riak operations.
## 
## Default: off
## 
## Acceptable values:
##   - on or off
## strong_consistency = on

## listener.http.<name> is an IP address and TCP port that the Riak
## HTTP interface will bind.
## 
## Default: 127.0.0.1:8098
## 
## Acceptable values:
##   - an IP/port pair, e.g. 127.0.0.1:10011
#listener.http.internal = 127.0.0.1:8098
## listener.protobuf.<name> is an IP address and TCP port that the Riak
## Protocol Buffers interface will bind.
## 
## Default: 127.0.0.1:8087
## 
## Acceptable values:
##   - an IP/port pair, e.g. 127.0.0.1:10011
#listener.protobuf.internal = 127.0.0.1:8087
listener.protobuf.internal = 127.0.0.1:10012

## The maximum length to which the queue of pending connections
## may grow. If set, it must be an integer > 0. If you anticipate a
## huge number of connections being initialized *simultaneously*, set
## this number higher.
## 
## Default: 128
## 
## Acceptable values:
##   - an integer
## protobuf.backlog = 128

## listener.https.<name> is an IP address and TCP port that the Riak
## HTTPS interface will bind.
## 
## Acceptable values:
##   - an IP/port pair, e.g. 127.0.0.1:10011
## listener.https.internal = 127.0.0.1:8098
listener.https.internal = 127.0.0.1:10011

## How Riak will repair out-of-sync keys. Some features require
## this to be set to 'active', including search.
## * active: out-of-sync keys will be repaired in the background
## * passive: out-of-sync keys are only repaired on read
## * active-debug: like active, but outputs verbose debugging
## information
## 
## Default: active
## 
## Acceptable values:
##   - one of: active, passive, active-debug
anti_entropy = passive 

## Specifies the storage engine used for Riak's key-value data
## and secondary indexes (if supported).
## 
## Default: bitcask
## 
## Acceptable values:
##   - one of: bitcask, leveldb, memory, multi
storage_backend = bitcask

## Controls which binary representation of a riak value is stored
## on disk.
## * 0: Original erlang:term_to_binary format. Higher space overhead.
## * 1: New format for more compact storage of small values.
## 
## Default: 1
## 
## Acceptable values:
##   - the integer 1
##   - the integer 0
object.format = 1

## Reading or writing objects bigger than this size will write a
## warning in the logs.
## 
## Default: 5MB
## 
## Acceptable values:
##   - a byte size with units, e.g. 10GB
object.size.warning_threshold = 5MB

## Writing an object bigger than this will send a failure to the
## client.
## 
## Default: 50MB
## 
## Acceptable values:
##   - a byte size with units, e.g. 10GB
object.size.maximum = 50MB

## Writing an object with more than this number of siblings will
## generate a warning in the logs.
## 
## Default: 25
## 
## Acceptable values:
##   - an integer
object.siblings.warning_threshold = 25

## Writing an object with more than this number of siblings will
## send a failure to the client.
## 
## Default: 100
## 
## Acceptable values:
##   - an integer
object.siblings.maximum = 100

## A path under which bitcask data files will be stored.
## 
## Default: $(platform_data_dir)/bitcask
## 
## Acceptable values:
##   - the path to a directory
bitcask.data_root = $(platform_data_dir)/bitcask

## Configure how Bitcask writes data to disk.
## erlang: Erlang's built-in file API
## nif: Direct calls to the POSIX C API
## The NIF mode provides higher throughput for certain
## workloads, but has the potential to negatively impact
## the Erlang VM, leading to higher worst-case latencies
## and possible throughput collapse.
## 
## Default: erlang
## 
## Acceptable values:
##   - one of: erlang, nif
bitcask.io_mode = erlang

## Set to 'off' to disable the admin panel.
## 
## Default: off
## 
## Acceptable values:
##   - on or off
riak_control = off

## Authentication mode used for access to the admin panel.
## 
## Default: off
## 
## Acceptable values:
##   - one of: off, userlist
riak_control.auth.mode = off

## If riak control's authentication mode (riak_control.auth.mode)
## is set to 'userlist' then this is the list of usernames and
## passwords for access to the admin panel.
## To create users with given names, add entries of the format:
## riak_control.auth.user.USERNAME.password = PASSWORD
## replacing USERNAME with the desired username and PASSWORD with the
## desired password for that user.
## 
## Acceptable values:
##   - text
## riak_control.auth.user.admin.password = pass

## This parameter defines the percentage of total server memory
## to assign to LevelDB. LevelDB will dynamically adjust its internal
## cache sizes to stay within this size.  The memory size can
## alternately be assigned as a byte count via leveldb.maximum_memory
## instead.
## 
## Default: 70
## 
## Acceptable values:
##   - an integer
leveldb.maximum_memory.percent = 70

## To enable Search set this 'on'.
## 
## Default: off
## 
## Acceptable values:
##   - on or off
search = off

## How long Riak will wait for Solr to start. The start sequence
## will be tried twice. If both attempts timeout, then the Riak node
## will be shutdown. This may need to be increased as more data is
## indexed and Solr takes longer to start. Values lower than 1s will
## be rounded up to the minimum 1s.
## 
## Default: 30s
## 
## Acceptable values:
##   - a time duration with units, e.g. '10s' for 10 seconds
search.solr.start_timeout = 30s

## The port number which Solr binds to.
## NOTE: Binds on every interface.
## 
## Default: 8093
## 
## Acceptable values:
##   - an integer
search.solr.port = 8093

## The port number which Solr JMX binds to.
## NOTE: Binds on every interface.
## 
## Default: 8985
## 
## Acceptable values:
##   - an integer
search.solr.jmx_port = 8985

## The options to pass to the Solr JVM.  Non-standard options,
## i.e. -XX, may not be portable across JVM implementations.
## E.g. -XX:+UseCompressedStrings
## 
## Default: -d64 -Xms1g -Xmx1g -XX:+UseStringCache -XX:+UseCompressedOops
## 
## Acceptable values:
##   - text
search.solr.jvm_options = -d64 -Xms1g -Xmx1g -XX:+UseStringCache -XX:+UseCompressedOops

EDIT 3

Doing a netstat to check on the port:

[idf@node1 riak]$ netstat -an | grep 10011
[idf@node1 riak]$ 

[idf@node1 ~]$  netstat -an | grep 10012
[idf@node1 ~]$

EDIT 4:

This is at the top of the crash file /var/log/riak/erl_crash.dump

=erl_crash_dump:0.2
Tue Apr  5 12:40:30 2016
Slogan: Kernel pid terminated (application_controller) ({application_start_failure,riak_core,{{shutdown,{failed_to_start_child,riak_core_
handoff_sup,{shutdown,{failed_to_start_child,riak_core_handoff_listen
System version: Erlang R16B02_basho8 (erts-5.10.3) [source] [64-bit] [smp:16:16] [async-threads:64] [kernel-poll:true] [frame-pointer]
Compiled: Thu Apr  2 16:53:43 2015

EDIT 5

Stopped firewall and tried again same error.

[idf@node1 ~]$ sudo systemctl stop firewalld
[idf@node1 ps ax | grep firewalld
 5485 pts/3    S+     0:00 grep --color=auto firewalld
[idf@node1 ~]$

Unsure how to fix?

Answer 1

I met the same problem, through the following method to solve the problem:

Edit the raik config file vim /etc/riak/riak.conf, Uncomment the line ## ssl.cacertfile = $(platform_etc_dir)/cacertfile.pem

## Acceptable values:
##   - the path to a file
   ssl.cacertfile = $(platform_etc_dir)/cacertfile.pem

## DTrace support Do not enable 'dtrace' unless your Erlang/OTP

Answer 2

{error,eaddrinuse} is coming for the handoff port which you have not changed and hence the default is 8099. In installed directory (/usr/lib64/riak) go to lib/riak-<version>/ebin and open riak_core.app or grep to see {handoff_port,8099} to confirm. Do

netstat -apn | grep 8099

to get the process name which is using the port. To change the handoff port you can add the following in the riak.conf in /etc/riak

handoff.port = 10013

or any other unused open port

Answer 3

This is the error that should concern you: {error,eaddrinuse}

It basically means:

EADDRINUSE — stands for error: address already in use. It means you have a server lis­ten­ing on the same port.

Check the riak configuration and ensure that nothing is running on the same IP:Port, netstat -an may help.

EDIT

It's crashing in the riak_core_handoff_listener.erl file. In particular, it's using the following configuration keys to open the IP/Port:

PortNum = app_helper:get_env(riak_core, handoff_port),
IpAddr = app_helper:get_env(riak_core, handoff_ip),

In your config you are setting something different (as far as I can see):

listener.http.internal = 127.0.0.1:10011
listener.protobuf.internal = 127.0.0.1:10012

Every time you change the config Riak will re-generate it. Here it shows you where are the config files read from:

-config /var/lib/riak/generated.configs/app.2016.04.05.11.55.10.config -args_file /var/lib/riak/generated.configs/vm.2016.04.05.11.55.10.args -vm_args /var/lib/riak/generated.configs/vm.2016.04.05.11.55.10.args

Please open the latest version and check exactly what configuration options are used. If riak_core,handoff_port and riak_core, handoff_ip are not configured there then it's likely using default values. According to the riak configuration document that would be port 8099. Either change that port or use netstat -an | grep 8099 to check what's listening on that port.