CODEWITHSUNDEEP
rsshscp
DadVlad
DadVlad

Reputation: 13

scp_download to download multiple files based on a pattern?

I need to download many files from a server (specifically tectia) ideally using the ssh package. These files all follow the a predictable pattern across multiple sub folders. The filepath is formatted like this

/directory/subfolder/A001/abcde001.csv

Where A001 counts up alongside the last 3 digits of the filename (/A002/abcde002.csv and so on)

In the vignette for scp_download it states that the files parameter may contain wildcards so I have tried to do something like

scp_download(session, "/directory/subfolder/A.*/abcde.*[.]csv", to=tempdir())

and

scp_download(session, "directory/subfolder/A\\d{3}/abcde\\d{3}[.]csv", to=tempdir())

but no matter which combination of patterns or wildcards I can think of (which isn't many) I only get something like

Warning: SSH warning: scp: /directory/subfolder/A\d{3}/abcde\d{3}[.]csv: No such file or directory

What I'm hoping to do is either find a way to do pattern matching here, or to find a way to store tectia directories as a string to be read by scp_download. I've made sure that my session is connected properly and it works without attempting to pattern match, which it does.

Upvotes: 0

Views: 631

Answers (1)

saeed
saeed

Reputation: 15

I had the same problem. The problem is that when you use * in your pattern it gets escaped when you send it to the server. However, when you request a special file name like this /directory/subfolder/A001/abcde001.csv, it works fine.

Finally I changed my code based on the below steps:

  1. I got the list of files/folders using ls command with ssh_exec_wait function and then store them on a variable.
  2. Download files in the variable separately
session <- ssh_connect("username@ip",passwd="password")
files<-capture.output(ssh_exec_wait(session, command = 'ls /directory/subfolder/A001/*'))
dnc1<- scp_download(session, files[1], to = paste0(getwd(),"/data/"))
dnc2<- scp_download(session, files[2], to = paste0(getwd(),"/data/"))
dnc3<- scp_download(session, files[3], to = paste0(getwd(),"/data/"))

The bottom 3 commands can be done in a loop as this could be hundreds or thousands of records.

Upvotes: 0

Related Questions