CODEWITHSUNDEEP
ssherlangelixir
Flame_Phoenix
Flame_Phoenix

Reputation: 17564

Connecting erlang observer to remote machine via public IP

Background

I have a machine in production running an elixir application (no access to iex, only to erl) and I am tasked with running an analysis on why we are consuming so much CPU. The idea here would be to launch observer, check the processes tab and see the processes with the most reductions.

How am I connecting?

To connect I am following a tutorial from a blog:

Their instructions are as follows:

  1. launch the app in the production machine with a cookie and a name
  2. from local run: ssh user@public_ip "epmd -names" to get the name of the app and the port used
  3. from local create a ssh tunnel to the remote machine: ssh -L 4369:user@public_ip:4369 -L 42877:user@public_ip:42877 user@public_ip (4369 is the epmd port by default, 42877 is the port of the app)
  4. from local connect to the remote machine using the node's name: erl -name "user@app_name" -setcookie "mah_cookie" -hidden -run observer

Problem

And now in theory I should be able to use observer on the machine. Instead however I am greeted with the following error:

Protocol ‘inet_tcp’: register/listen error: epmd_close

So, after scouring the dark side of internet, I decided to use sudo journalctl -f to check all the logs of the machine and I found this:

channel 3: open failed: administratively prohibited: open failed
my_app_name sshd[8917]: error: connect_to [email protected]: unknown host (Name or service not known)
/scripts/watchdog.sh")
my_app_name CRON[9985]: pam_unix(cron:session): session closed for user flame

Where:

  1. erlang -name: my_app_name
  2. machine user: flame
  3. machine public ip: 99.999.99.999 (obviously not real)

so it tells me, unknown host ?? I am confused since 99.999.99.999 is the public IP of the machine itself!

Questions

  1. What am I doing wrong?
  2. I read that in older versions of erlang I can’t monitor a machine with observer if they are in different networks (which is the case, because I want to monitor this machine from my localhost) but I didn’t find any information regarding this in modern days.
  3. If this is in fact impossible, what alternatives do I have?

Upvotes: 4

Views: 1989

Answers (1)

Flame_Phoenix
Flame_Phoenix

Reputation: 17564

Solution

After 3 days of non-stop searching, I finally found something that works. To summarize I am putting it here everything I did.

All steps in local machine:

  1. get the ports from the remote server:
> ssh remote-user@remote-ip "epmd -names"
epmd: up and running on port 4369 with data:
name super_duper_app at port 43175
  1. create a ssh tunel with the ports:

ssh remote-user@remote-ip -L4369:localhost:4369 -L43175:localhost:43175

  1. On another terminal in your local machine, run a iex terminal with the cookie the app in your remote server is using. Then connect to it and start observer:
iex --name [email protected] --cookie super_duper_cookie
Node.connect :"[email protected]"
> true
:observer.start

With observer started, select the machine from the Nodes menu.

Possible setbacks

If you have tried this and it didn't work there are a few things you can check for:

  1. Check if the EPMD port on your local machine is free, if not, kill the process using it and free it.
  2. Check your ssh tunneling keys and configurations for permissions. As @Roberto Aloi pointed out this link can be useful: https://unix.stackexchange.com/questions/14160/ssh-tunneling-error-channel-1-open-failed-administratively-prohibited-open

Upvotes: 4

Related Questions