CODEWITHSUNDEEP
linuxmatlabsshremote-access
tiswas
tiswas

Reputation: 2101

Reading files on other machine with matlab script

I have a huge bunch of files on another machine (linux) that I normally access via ssh (or scp if I want to transfer, of course). I also have lots of matlab scripts on my own machine (mac os x) that, as far as I know, can not be run on the other machine easily.

Ideally what I'd like to do is run the scripts from my machine, but on the data on the other machine. Is it possible to do this kind of thing via ssh or something similar? Or am I best off just scp-ing the (really very big) bunch of files onto my own machine and then doing the analysis from there.

Cheers in advance, Adam

Upvotes: 1

Views: 1782

Answers (2)

mnagel
mnagel

Reputation: 6854

I see two possibilities here:

  1. Before execution, pull the data to your local computer using rsync.

  2. Mount the remote data locally with sshfs and let the data transfer happen implicitly.

Depending on the criteria you optimize for (local disk usage, amount of data transferred, latency/throughput, amount of manual steps, ...) you might want to pick one or the other.

Upvotes: 2

High Performance Mark
High Performance Mark

Reputation: 78316

Perhaps an extended comment rather than an answer:

When your Matlab code runs the data it needs has to be in the same RAM as the code. This remains true wherever the files which store the data persistently are stored. There are basically three options:

  1. Copy your Matlab code to the machine where the data is stored and run the analyses in the RAM of the remote machine. This is likely to minimise the volume of data transferred (copy the Matlab program to the remote machine) but you may have to pull results back to your local machine. Your question suggests that you may not be able to run Matlab on the remote machine so this may not be an option for you.

  2. Copy the data, one file at a time, from the remote machine to the local machine as each program runs. This will result in relatively-many, relatively-small transfers of data.

  3. Copy the data, in one lump, from the remote machine to the local machine, and run all your programs locally.

You can probably see where I'm going on this: to minimise total execution time you should minimise data transfer time (ie latency (per-file time) and bandwidth (per-byte time)) with option 3 than option 2. So yes, I think your best option is to scp the whole bunch of files from remote to local.

Upvotes: 2

Related Questions