Remote Tensorboard not working with SSH tunnel

Question

From home, I am trying to monitor a tensorflow training with Tensorboard which is running on my machine at work.
We have an ssh gateway to access to machines at work, so I have to make an ssh tunnel, and from what I saw on internet this is how to do it to connect with the default port of Tensorboard 6006:

ssh -NfL 6006:remote_machine:6006 user@ssh_gateway_machine

Then on the remote machine:

tensorboard --logdir dir/ --port 6006

Then on my machine at home:

http://localhost:6006

But the page is white and loading forever, it cannot retrieve the data.

When I am trying to monitor this same machine but from another machine at work, using port forwarding the same way (without ssh tunnel) works fine, so the problem is not comming from Tensorboard rather from the ssh tunnel I guess, but I cannot figure out why.

Answer 1

In my case, I was running tensorboard on ubuntu using tensorboard --logdir=runs and I was getting the forever loading screen at http://localhost:6008

I had ran the command several times and every time it had spun a new port so the port that I was trying to access was being used by an orphan tensorboard leading to the forever loading screen.

What ended up working for me was

kill -9 <pid_my_old_tensorboards>

and running a fresh tensorboard using

tensorboard --logdir=runs --port 6008

Answer 2

I agree, you have to execute with bind_all option :

tensorboard --logdir dir/ --port 6006 --bind_all

But, maybe in your case, you have to just check if the ssh_gateway_machine ssh server is configured to allow port forwarding : AllowTcpForwarding yes ?

Explanations here :

• https://www.ssh.com/ssh/tunneling/example
• https://www.ssh.com/ssh/sshd_config/

Example :

ssh user@ssh_gateway_machine
sudo -i
vi /etc/ssh/sshd_config

If possible (ask your company IT security staff), edit lines :

AllowTcpForwarding yes

But in my case, instead of -L option executed locally, I had to make a ssh remote port forwarding -R option executed remotely (on remote_machine) because I cannot access from internet directly to any ports of remote_machine network.

So, I think my -R remote port forwarding method is better, if your remote_machine have access to internet and you can install/activate a ssh server (on a NAS or on your local_machine).

So, I repeat, it is done with : -R option executed on remote_machine.

So, the ssh_gateway_machine become local (Synology NAS for instance).

If you don't have a NAS, maybe it could be your local_machine if you install/activate a SSH server. (ssh_gateway_machine == local_machine)

And you have to modify Port forwarding of your internet provider box/ network router) to forward ssh incoming port to your internal ssh_gateway_machine (to be able to have incoming access from internet through your internet provider box up to your ssh_gateway_machine.

You have to check and modify server ssh config in the ssh_gateway_machine (Synology NAS),GatewayPorts & AllowTcpForwarding options :

ssh user@ssh_gateway_machine
sudo -i
vi /etc/ssh/sshd_config

Edit lines :

GatewayPorts yes
AllowTcpForwarding yes

Also, don't forget to modify Firewall rules of ssh_gateway_machine to accept only ssh incoming access from remote_machine public IP. Maybe it can also be done on your internet router.

After, you have to execute from remote_machine :

ssh user@ssh_gateway_machine_public_ip -R 6006:localhost:6006

with ssh_gateway_machine_public_ip = local_machine_public_ip = internet_box public_ip

Picture of the network : My network map

Answer 3

Try using bind_all parameter while launching the tensorboard. It allows remote connection for tensorboard. So your command should be like

tensorboard --logdir dir/ --port 6006 --bind_all

reference : https://github.com/tensorflow/tensorboard/blob/master/README.md#i-get-a-network-security-popup-every-time-i-run-tensorboard-on-a-mac