How to get list of files from an SFTP server?

Question

I have a problem and hoping to get a solution. I also have written some code but it needs some modification.

Problem: I have a SFTP server (for privacy purposes I will give dummy credentials) that I need to connect to.

Server name: server-name
port: 22
username: username
password: password

When I connect to the server, it automatically drops me in the /FGV directory. inside this directory are couple other folders. I need to grab a LIST of xml messages from the /FGV/US/BS/ directory and place them in a LIST (files in the form of File). In the list, I need to have the directory of the file, file name and the file body. I was thinking of creating an object and putting this information in there and creating a List of that object.

My current code creates a connection and downloads only ONE xml file. If there are two xml files, then the file in my local machine has nothing as content.

import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.OutputStream;

import com.jcraft.jsch.Channel;
import com.jcraft.jsch.ChannelSftp;
import com.jcraft.jsch.JSch;
import com.jcraft.jsch.Session;

public class SFTPinJava {

    public SFTPinJava() {
    }

    public static void main(String[] args) {
        String SFTPHOST = "server-name";
        int SFTPPORT = 22;
        String SFTPUSER = "username";
        String SFTPPASS = "password";
        String SFTPWORKINGDIR = "/FGV";

        Session session = null;
        Channel channel = null;
        ChannelSftp channelSftp = null;

        try {
            JSch jsch = new JSch();
            session = jsch.getSession(SFTPUSER, SFTPHOST, SFTPPORT);
            session.setPassword(SFTPPASS);
            java.util.Properties config = new java.util.Properties();
            config.put("StrictHostKeyChecking", "no");
            session.setConfig(config);
            session.connect();
            channel = session.openChannel("sftp");
            channel.connect();
            channelSftp = (ChannelSftp) channel;
            channelSftp.cd(SFTPWORKINGDIR);
            byte[] buffer = new byte[1024];
            BufferedInputStream bis = new BufferedInputStream(
                channelSftp.get("/FGV/US/BS/FGVCustomsEntryLoaderService.xml"));
            File newFile = new File(
                "C:\\workspace\\Crap\\src\\org\\raghav\\stuff\\XML_FROM_SERVER.xml");
            OutputStream os = new FileOutputStream(newFile);
            BufferedOutputStream bos = new BufferedOutputStream(os);
            int readCount;
            //System.out.println("Getting: " + theLine);
            while ((readCount = bis.read(buffer)) > 0) {
                //System.out.println("Writing: ");
                bos.write(buffer, 0, readCount);
            }
            
            while(session != null){
                System.out.println("Killing the session");
                session.disconnect();
                bis.close();
                bos.close();
                System.exit(0);
            }
            
        } catch (Exception ex) {
            ex.printStackTrace();
        }
    }
}

I need to change this code so that it would grab multiple files and puts them in a list of objects. that object should have the directory of the file, the file name and the body of the file.

Answer 1

This code is working well.

        @SuppressWarnings("null")
        Vector<ChannelSftp.LsEntry> filesList = channelSftp.ls("*.txt");
        logger.info("filesList size:" + filesList.size()); 
        if(filesList != null) {
            for(ChannelSftp.LsEntry entry : filesList) {
                InputStream stream = channelSftp.get(entry.getFilename());
                br = new BufferedReader(new InputStreamReader(stream));
                if(br != null) {
                    while ((line = br.readLine()) != null) {
                    }
                }
           }
        }

Answer 2

you can list all files in the given directory using

Vector<ChannelSftp.LsEntry> list = channelSftp.ls("*.csv");
for(ChannelSftp.LsEntry entry : list) {
     System.out.println(entry.getFilename()); 
}

add this code after

channelSftp.cd(SFTPWORKINGDIR);

now you'll get list of file objects. file object is entry.if you want to download all files . add this code inside to for loop.

byte[] buffer = new byte[1024];
BufferedInputStream bis = new BufferedInputStream(channelSftp.get(entry.getFilename()));
File newFile = new File("C:/Users/Desktop/sftpStuff/"+entry.getFilename());
OutputStream os = new FileOutputStream(newFile);
BufferedOutputStream bos = new BufferedOutputStream(os);
int readCount;
//System.out.println("Getting: " + theLine);
while( (readCount = bis.read(buffer)) > 0) {
  System.out.println("Writing: "+entry.getFilename() );
  bos.write(buffer, 0, readCount);
}
bis.close();
bos.close();

Answer 3

ChannelSftp has a method to obtain a list of files from a path. I'm assuming the files you want to download are all in the in the same directory and that you would do:

channelSftp.cd("/FGV/US/BS/");
Vector ls = channelSftp.ls("*")

http://epaul.github.io/jsch-documentation/simple.javadoc/com/jcraft/jsch/ChannelSftp.html#ls(java.lang.String)

Once you have that list (a Vector is returned), set up a for loop or iterator to use each ChannelSftp.LsEntry in the Vector, using the getFilename method to obtain the filename. You can check to make sure it has an .xml extension. Everything from the byte[] buffer line on in your code goes inside the loop. You'll need to add successive reads to your buffer to a StringBuffer if you want to preserve the contents of the file.

As for storing the filename, contents of each, and directory, you could use a simple wrapper. Example:

public class SFTPFileList {
    public String directory;
    public ArrayList<String> filenames;
    public ArrayList<StringBuffer> fileContents;  

    /**
     *Simple method to add a filename and its contents.
     *filenames and fileContents will have the same index
     *for filename<->content. 
     **/
    public void addFile(String filename, StringBuffer content) {
        filenames.add(filename);
        fileContents.add(content);
    }
}

You can add methods, constructor, etc. to properly get, set. Or access directly. For the latter, outside your loop:

SFTPFileList sftpFileList = new SFTPFileList();
sftpFileList.directory = "/FGV/US/BS/";
ArrayList<String> fileNames = new ArrayList<>();
ArrayList<String,StringBuffer> contents = new ArrayList<>();
sftpFileList.filenames = filenames;
sftpFileList.contents = contents;

Inside your loop:

ChannelSftp.LsEntry entry = iter.next();  //or from a for loop
String fileName = entry.getFileName();
StringBuffer fileContent = new StringBuffer(); //this where you will add the results of each read -> buffer 
...
read content
...
sftpFileList.addFile(fileName,fileContent);

Then outside of the loop you'll have access to all the filenames and their content via another method or methods that you can write. Or directly access those members as necessary.

Answer 4

This is a way to list files of destination directory.

Vector filelist = channelSftp.ls(SFTPWORKINGDIR);
for(int i=0; i<filelist.size();i++){
     System.out.println(filelist.get(i).toString());
     // Grap and get the file by WORKING_DIR/filelist.get(i).toString();
     // Save it to your local directory with its original name. 

}

Second, while in for loop you can download all files if they are requested xml files those you need.