CODEWITHSUNDEEP
pythonpython-3.xsftppysftp
defiant
defiant

Reputation: 3341

Obtaining attributes of files and folders iterated with pysftp Connection.walktree?

I am trying to list files and its modification time recursively under a directory using pysftp. The problem is, it shows file doesn't exist for all files, but not for directories.

Here is my code:

class SftpOps:
    def __init__(self):
        config = ConfigManager()
        host = config.getSftpUrl()
        port = config.getSftpPort()
        user = config.getSftpUsername()
        password = config.getSftpPassword()
        self.source = config.getSourceDirRelativePath()

        cnopts = pysftp.CnOpts()
        cnopts.hostkeys.load('host_key')

        self.sftp = pysftp.Connection(
            host, port=port, username=user, password=password, cnopts=cnopts)

    def downloadSourceDir(self):
        print(self.sftp.listdir())
        self.sftp.walktree(self.source, self.fileModifiedTimeCheck,
                           self.dirModifiedTimeCheck, self.fileModifiedTimeCheck, recurse=True)
        self.sftp.close()

    def fileModifiedTimeCheck(self, filepath):
        filepath = os.path.join(self.source, filepath)
        try:
            for attr in self.sftp.listdir_attr(filepath):
                print(f"{filepath}: {attr.st_atime}")
        except FileNotFoundError as err:
            print(f"No file at: {filepath}, failed with err: {err}")
        except OSError as err:
            print("OS error: {0}".format(err))
        except:
            print("Unexpected error:", sys.exc_info()[0])
            raise

    def dirModifiedTimeCheck(self, filepath):
        filepath = os.path.join(self.source, filepath)
        try:
            for attr in self.sftp.listdir_attr(filepath):
                print(f"{filepath}: {attr.st_atime}")
            filepath = "tmp/"+filepath
        except FileNotFoundError:
            print(f"No dir at: {filepath}")
        except OSError as err:
            print("OS error: {0}".format(err))
        except:
            print("Unexpected error:", sys.exc_info()[0])
            raise


# class LogOps:
#     def __init__(self):


# class EmailOps:
#     def __init__(self):

print("=========================================")
test_obj = SftpOps()
test_obj.downloadSourceDir()
print("=========================================")

When I try this for a directory with the below structure enter image description here

It gives the errors as:

=========================================
['.DS_Store', 'about-me.html', 'favicon.ico', 'index.html', 'test']
No file at: /Downloads/techtuft/.DS_Store, failed with err: [Errno 2] No such file
No file at: /Downloads/techtuft/about-me.html, failed with err: [Errno 2] No such file
No file at: /Downloads/techtuft/favicon.ico, failed with err: [Errno 2] No such file
No file at: /Downloads/techtuft/index.html, failed with err: [Errno 2] No such file
/Downloads/techtuft/test: 1569165379
No file at: /Downloads/techtuft/test/style.css, failed with err: [Errno 2] No such file
=========================================

Please note, how it doesn't show error for the directory "test".

Upvotes: 2

Views: 2246

Answers (2)

Martin Prikryl
Martin Prikryl

Reputation: 202494

You cannot use Connection.listdir_attr for files.

You should use Connection.stat.

Though that would be terribly inefficient anyway. You better copy over the implementation of walktree function and make it call Connection.listdir_attr instead of Connection.listdir. That way you obtain timestamps for all files in the directory in a single call to the server, en masse, instead of retrieving them inefficiently file-by-file.

See also Python SFTP download files older than x and delete networked storage.

Upvotes: 1

BernardL
BernardL

Reputation: 5454

So it seems that the method listdir_attr displays current entries in a directory but does not handle special entries such as paths containing . and ... Make reference here from the docs. This is why the test folder was not included into your error message.

You can however use the lstat method to identify files:

>>> for i in sftp.listdir():
...     lstatout=str(sftp.lstat(i)).split()[0]
...     if 'd' not in lstatout: #do something
...

Upvotes: 0

Related Questions