Reputation: 1384

List first N files in SFTP with JSch

I need to get a recursive list of file names in an SFTP folder. Now I'm using the following code:

private void getFilesRecursive(List<LsEntry> fileList, List<FileDto> response, String dirParentName,
        SftpManagerWithPool manager) throws SftpException {
    for (LsEntry file : fileList) {
        if (!file.getAttrs().isDir()) {
            response.add(new FileDto(file.getFilename(), StorageType.FOLDER.getName(),
                    new Attributes(file.getAttrs().getATime(), file.getAttrs().getMTime(),
                            file.getAttrs().getAtimeString(), file.getAttrs().getMtimeString(),
                            file.getAttrs().getPermissionsString(), getPathWitoutRootDirectoryt(dirParentName),
                            file.getAttrs().getSize(), file.getAttrs().isDir())));
        } else if (file.getAttrs().isDir() && !".".equals(file.getFilename())) {
            List<LsEntry> files = manager.listFiles(context.getBasePath().concat("/")
                    .concat(dirParentName.concat("/").concat(file.getFilename())));
            getFilesRecursive(files, response, dirParentName.concat("/").concat(file.getFilename()), manager);
        }
    }
}

public List<FileDto> getFiles() throws ServiceException {
    Session session = null;
    SftpManagerWithPool manager = null;
    try {
        session = getSession();
        manager = new SftpManagerWithPool(session);
        manager.connect();
        List<FileDto> response = new ArrayList<>();
        List<LsEntry> files = manager
                .listFiles(context.getBasePath().concat("/").concat(context.getPathToProcess()));
        getFilesRecursive(files, response, context.getPathToProcess(), manager);
        return response;
    } catch (Exception e) {
        throw new ServiceException(HttpStatus.INTERNAL_SERVER_ERROR.value(), e.getMessage(),
                e.getLocalizedMessage());
    } finally {
        if (manager != null) {
            manager.disconnect();
            try {
                returnSession(session);
            } catch (Exception e) {
                throw new ServiceException(HttpStatus.INTERNAL_SERVER_ERROR.value(), e.getMessage(),
                        e.getLocalizedMessage());
            }
        }
    }
}

The following is the method in SftpManagerWithPool :

@SuppressWarnings("unchecked")
public List<LsEntry> listFiles(String pathFrom) throws SftpException {
    if (c == null || session == null || !session.isConnected() || !c.isConnected()) {
        throw new SftpException(ErrorCode.LIST_FILES.ordinal(), "Connection to server is closed. Open it first.");
    }
    String filePath = pathFrom + "/";
    return c.ls(filePath);
}

Everything works well with no more than 5k files but when we have more files it takes a lot of time and it results in timeouts.

My question is, how can I use the c.ls(filePath); method to list only the first N files in the folder? I'm looking for something similar to the Linux shell command ls -U | head -4

----EDIT-----

I modified my method listFiles as follow but I still don't get a better performance:

public List<LsEntry> listFiles(String pathFrom, int top) throws SftpException {
        if (c == null || session == null || !session.isConnected() || !c.isConnected()) {
            throw new SftpException(ErrorCode.LIST_FILES.ordinal(), "Connection to server is closed. Open it first.");
        }
        String filePath = pathFrom + "/";
        List<LsEntry> response = new ArrayList<>();
        c.ls(filePath, new LsEntrySelector() {

            @Override
            public int select(LsEntry record) {
                if (response.size() <= top) {
                    response.add(record);
                    return LsEntrySelector.CONTINUE;
                } else {
                    return LsEntrySelector.BREAK;
                }
            }
        });
        return response;
    }

Thank you so much :)

Upvotes: 3

Answers (2)

Devraj Reddy

Reputation: 1

Below code you can read N number of files.

Vector<LsEntry> files = new Vector<LsEntry>();
channelSftp.ls(path + Constants.XML_FORMAT,new ChannelSftp.LsEntrySelector() {
public int select(ChannelSftp.LsEntry lsEntry) {
files.add(lsEntry);
log.debug("lsEntry is {}", lsEntry.getFilename());
if (files.size() >= maxFileCount) {
log.debug("file size is {}", files.size());
return ChannelSftp.LsEntrySelector.BREAK;
}
return ChannelSftp.LsEntrySelector.CONTINUE;
}
});

Upvotes: 0

Martin Prikryl

Reputation: 202652

Use this the overload of ChannelSftp.ls method that takes LsEntrySelector interface:

public void ls(String path, LsEntrySelector selector)

Implement the LsEntrySelector.select to collect the file entries. Once you have enough of them, make it return BREAK.

Upvotes: 2

List first N files in SFTP with JSch

Answers (2)

Related Questions