Reputation: 415
From home, I am trying to monitor a tensorflow training with Tensorboard which is running on my machine at work.
We have an ssh gateway to access to machines at work, so I have to make an ssh tunnel, and from what I saw on internet this is how to do it to connect with the default port of Tensorboard 6006:
ssh -NfL 6006:remote_machine:6006 user@ssh_gateway_machine
Then on the remote machine:
tensorboard --logdir dir/ --port 6006
Then on my machine at home:
http://localhost:6006
But the page is white and loading forever, it cannot retrieve the data.
When I am trying to monitor this same machine but from another machine at work, using port forwarding the same way (without ssh tunnel) works fine, so the problem is not comming from Tensorboard rather from the ssh tunnel I guess, but I cannot figure out why.
Upvotes: 2
Views: 3247
Reputation: 7207
In my case, I was running tensorboard on ubuntu using tensorboard --logdir=runs
and I was getting the forever loading screen at http://localhost:6008
I had ran the command several times and every time it had spun a new port so the port that I was trying to access was being used by an orphan tensorboard leading to the forever loading screen.
What ended up working for me was
kill -9 <pid_my_old_tensorboards>
and running a fresh tensorboard using
tensorboard --logdir=runs --port 6008
Upvotes: 1
Reputation: 134
I agree, you have to execute with bind_all option :
tensorboard --logdir dir/ --port 6006 --bind_all
But, maybe in your case, you have to just check if the ssh_gateway_machine
ssh server is configured to allow port forwarding : AllowTcpForwarding yes
?
Explanations here :
Example :
ssh user@ssh_gateway_machine
sudo -i
vi /etc/ssh/sshd_config
If possible (ask your company IT security staff), edit lines :
AllowTcpForwarding yes
But in my case, instead of -L
option executed locally, I had to make a ssh remote port forwarding -R
option executed remotely (on remote_machine
) because I cannot access from internet directly to any ports of remote_machine
network.
So, I think my -R
remote port forwarding method is better, if your remote_machine
have access to internet and you can install/activate a ssh server (on a NAS or on your local_machine
).
So, I repeat, it is done with : -R
option executed on remote_machine
.
So, the ssh_gateway_machine
become local (Synology NAS for instance).
If you don't have a NAS, maybe it could be your local_machine
if you install/activate a SSH server. (ssh_gateway_machine
== local_machine
)
And you have to modify Port forwarding of your internet provider box/ network router) to forward ssh incoming port to your internal ssh_gateway_machine
(to be able to have incoming access from internet through your internet provider box up to your ssh_gateway_machine
.
You have to check and modify server ssh config in the ssh_gateway_machine
(Synology NAS),GatewayPorts
& AllowTcpForwarding
options :
ssh user@ssh_gateway_machine
sudo -i
vi /etc/ssh/sshd_config
Edit lines :
GatewayPorts yes
AllowTcpForwarding yes
Also, don't forget to modify Firewall rules of ssh_gateway_machine
to accept only ssh incoming access from remote_machine
public IP.
Maybe it can also be done on your internet router.
After, you have to execute from remote_machine
:
ssh user@ssh_gateway_machine_public_ip -R 6006:localhost:6006
with ssh_gateway_machine_public_ip
= local_machine_public_ip
= internet_box public_ip
Picture of the network : My network map
Upvotes: 3
Reputation: 608
Try using bind_all
parameter while launching the tensorboard. It allows remote connection for tensorboard. So your command should be like
tensorboard --logdir dir/ --port 6006 --bind_all
Upvotes: 6