Jepsen : SSH issue

Question

Background:

I am trying to run the Jepsen (https://github.com/jepsen-io/jepsen) on Ubuntu 16.04. I have created 6 virtual machines (on my windows 10 pc) and replaced the host name by n1, n2, .... n6. The user name and password is same for all VMs.

Now, from any VM, I can directly ssh to any other VM. ( For example, from n6, I can just execute this command "ssh n1" and it works without any problem). I am trying to run the node jepsen tool from node 6 ( hostname n6 ) so that it can create the db in node n1,n2,n3,n4 and n5.

So far I have not made any changes in the jepsen code. I have compiled it using "lein install" command and now trying to run it.

Problem:

To simply run the test, I followed the instruction provided in the jepsen's webpage ( "cd aerospike"; "lein test" ) But it is giving me the following error:

user@Ubuntu6:~/Desktop/jepsen_cassandra/aerospike$ lein test

lein test aerospike.core-test
jepsen.generator$concat$reify__7285@577dac16>

lein test :only aerospike.core-test/counter

ERROR in (counter) (Session.java:781)
Uncaught exception, not in assertion.
expected: nil
actual: com.jcraft.jsch.JSchException: reject HostKey: n1 at com.jcraft.jsch.Session.checkHost (Session.java:781)
com.jcraft.jsch.Session.connect (Session.java:342)
com.jcraft.jsch.Session.connect (Session.java:183)
clj_ssh.ssh$eval6796$fn__6803.invoke (ssh.clj:118)
clj_ssh.ssh.protocols$eval6722$fn__6745$G__6713__6754.invoke (protocols.clj:4)
clj_ssh.ssh$connect.invoke (ssh.clj:401)
jepsen.control$session.invoke (control.clj:197)
clojure.lang.AFn.applyToHelper (AFn.java:154)
clojure.lang.AFn.applyTo (AFn.java:144)
clojure.core$apply.invoke (core.clj:624)
clojure.core$with_bindings_STAR_.doInvoke (core.clj:1862)
clojure.lang.RestFn.applyTo (RestFn.java:142)
clojure.core$apply.invoke (core.clj:628)
clojure.core$bound_fn_STAR_$fn__4140.doInvoke (core.clj:1884)
clojure.lang.RestFn.applyTo (RestFn.java:137)
clojure.core$apply.invoke (core.clj:624)
jepsen.core$fcatch$wrapper__8355.doInvoke (core.clj:55)
clojure.lang.RestFn.invoke (RestFn.java:408)
clojure.core$pmap$fn__6328$fn__6329.invoke (core.clj:6466)
clojure.core$binding_conveyor_fn$fn__4145.invoke (core.clj:1910)
clojure.lang.AFn.call (AFn.java:18)
java.util.concurrent.FutureTask.run (FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617)
java.lang.Thread.run (Thread.java:748)
jepsen.generator$concat$reify__7285@254210b1>

lein test :only aerospike.core-test/cas-register

ERROR in (cas-register) (Session.java:781)
Uncaught exception, not in assertion.
expected: nil
actual: com.jcraft.jsch.JSchException: reject HostKey: n1
at com.jcraft.jsch.Session.checkHost (Session.java:781)
com.jcraft.jsch.Session.connect (Session.java:342)
com.jcraft.jsch.Session.connect (Session.java:183)
clj_ssh.ssh$eval6796$fn__6803.invoke (ssh.clj:118)
clj_ssh.ssh.protocols$eval6722$fn__6745$G__6713__6754.invoke (protocols.clj:4)
clj_ssh.ssh$connect.invoke (ssh.clj:401)
jepsen.control$session.invoke (control.clj:197)
clojure.lang.AFn.applyToHelper (AFn.java:154)
clojure.lang.AFn.applyTo (AFn.java:144)
clojure.core$apply.invoke (core.clj:624)
clojure.core$with_bindings_STAR_.doInvoke (core.clj:1862)
clojure.lang.RestFn.applyTo (RestFn.java:142)
clojure.core$apply.invoke (core.clj:628)
clojure.core$bound_fn_STAR_$fn__4140.doInvoke (core.clj:1884)
clojure.lang.RestFn.applyTo (RestFn.java:137)
clojure.core$apply.invoke (core.clj:624)
jepsen.core$fcatch$wrapper__8355.doInvoke (core.clj:55)
clojure.lang.RestFn.invoke (RestFn.java:408)
clojure.core$pmap$fn__6328$fn__6329.invoke (core.clj:6466)
clojure.core$binding_conveyor_fn$fn__4145.invoke (core.clj:1910)
clojure.lang.AFn.call (AFn.java:18)
java.util.concurrent.FutureTask.run (FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617)
java.lang.Thread.run (Thread.java:748)

Ran 2 tests containing 2 assertions.
0 failures, 2 errors.
Tests failed.

First I tried to run the code as it is. But later I have added the username and password of the 5 VMs ( that is required to ssh ) in Line 21 and 22 of ( https://github.com/jepsen-io/jepsen/blob/master/jepsen/src/jepsen/control.clj ). But it did not solve the problem.

I will highly appreciate any solution/guideline to solve this problem.

Answer 1

this line

actual: com.jcraft.jsch.JSchException: reject HostKey: n1 at com.jcraft.jsch.Session.checkHost

indicates that the system initiating the ssh connection is unwilling to trust the identity of the computer it's connecting to. So it's not a problem with passwords/keys not allowing you in, rather it's the opposite, the system starting the connection is not sure of the identity of the system it's connecting to.

The most common cause is that you have connected to a system in the past at the same IP or hostname which has since been replaced with this system. SSH guarantees that if a different computer responds to the same name, it will set off the alarms and not connect rather than risk a man-in-the-middle attack. This results in false positives when a system really has been replaced.

Try sshing from this system to the target host and make sure it work. if not try removing that system's IP from ~/.ssh/known_hosts if it's there. also consider if anything in your ~/.ssh/config file would be affecting this host, perhaps by requiring it to have a specific host key for instance

---

since it doesn't seem to be a host key conflict the next thing to check is what kind of host key the hosts are using. Jsch is it's own implementation of ssh that supports a different set of keys than openSSH. you can determine the system's host key type by running

ssh -vvv hostname

...
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256

then google for jsch and that host key algorithm and see if there are any open issues on it.