CODEWITHSUNDEEP
linuxshelldateftpget
Dhruuv
Dhruuv

Reputation: 343

Get all the files from a FTP location with previous week's dates in the file names using Linux

I have a weird requirement where I have to get the files from a FTP(Lets say FTP1) location and place it on my current FTP(Lets say FTP2) location. The issue is, these are daily files (in a pattern Sales_YYYYMMDD_report.csv) and are placed every day on FTP1 and my process usually runs on Monday(eg. 09-Sept-2013) which has to use the file of the previous week starting from Sunday(eg. 01-Sept-2013) to Saturday(eg. 07-Sept-2013) place them on FTP2 location and then run the Informatica process. For an instance, if I run the process on Monday,09-Sept-2013, I have to pull all the files from FTP1 which have file names such as 

Sunday file --> Sales_20130901_report.csv  
Monday file --> Sales_20130902_report.csv  
Tuesday file --> Sales_20130903_report.csv  
Wednesday file --> Sales_20130904_report.csv  
Thursday file --> Sales_20130905_report.csv  
Friday file --> Sales_20130906_report.csv  
Saturday file --> Sales_20130907_report.csv

How can I achieve this in a shell script? I know the part to get the files from another FTP, but I am not sure how to get the 7 files.

P.S: I cannot use the file creation/last modified timestamps to get the files. Irrespective of the created timestamp and the day I run my Informatica process, I have to get the files which have last week's dates in the file names and put in my FTP2 location and then continue with them.

Please help...

Upvotes: 0

Views: 1422

Answers (2)

Floris
Floris

Reputation: 46365

The following script should have all the elements you need:

#!/bin/bash
# assuming this is run from the directory where you want the files
# to end up

function getIt {
  echo "ftp-ing " $1
  # here comes the heart of the ftp session
  ftp -inv << _EOF_
  open home.machine.com
  user myname mypassword
  cd /the/path/where/the/file/lives
  get $1
  bye
_EOF_
  }

# generate the seven file names for the previous seven days:
for d in {1..7}
  do
    theCmd="/bin/date -v -"$d"d +%Y%m%d"
    theDate=`$theCmd`
    fileName="Sales_"$theDate"_report.csv"
    getIt $fileName
  done

It should be mostly self-explanatory: but note in particular that the end of the heredoc (the "encapsulated ftp script", if you like) MUST be at the start of the line, with no white space before it, and no white space after. Also - depending on how important security on this machine is you may want to do something different with the password; maybe you even want to use something other than "vanilla" ftp. But I think this should get you going in the right direction.

Tested up to the actual ftp bit itself... when I comment that out the script correctly calls getIt() seven times, with a string representing the filename you specified. Obviously I can't easily test the ftp...

Note also this will just copy to the directory where you start the script from; if you need it to end up somewhere else you may need a second ftp script. I'm sure you can figure that out though.

Upvotes: 1

Mehul Rathod
Mehul Rathod

Reputation: 1244

you can use following command in linux (tested on Cent OS 6), change the -1 day for appropriate dates

yesterday="date +%Y%m%d --date="-1 day""

more reference = http://blog.midnightmonk.com/85/bash/bash-date-manipulation.shtml

Upvotes: 0

Related Questions