CODEWITHSUNDEEP
pythonpython-3.xsshsftppysftp
user3751069
user3751069

Reputation: 111

How do i monitor the progress of a file transfer through pysftp

I'm using python 3.3.2 and pysftp to make a back-up utility that stores copies of files on another computer on my network.

I already know how to use pysftp to transfer theses files however i would like to see the progress of the transfer (bytes per second, percent completion, time remaining) instead of some static print statement indicating that the transfer has started or finished

something like this maybe ( filename: 3.4MiB/s | 40%<#########=============> | 0:03:54 remaining )

Edit: I could probably build a progress bar if i just knew how to get pysftp's put command to yeild the information, THAT is what i want to know how to do

Edit: I think i found my answer on another question after extensive digging:

How to see (log) file transfer progress using paramiko?

Upvotes: 8

Views: 11789

Answers (6)

mivilar
mivilar

Reputation: 151

This is what it would look like using rich.

import pysftp
from rich.progress import Progress

with pysftp.Connection(host=None, username=None, password=None) as sftp:
    file_to_download = '.'
    with Progress() as progress:
        task = progress.add_task(f"[red]Downloading {file_to_download} ...", total=None)

        sftp.get(
            file_to_download,
            local_path,
            callback=lambda so_far, total: progress.update(task, completed=so_far, total=total),
        )

Upvotes: 1

Izaak Cornelis
Izaak Cornelis

Reputation: 322

If you want to use the tqdm progressbar:

from tqdm import tqdm
import pysftp

class Progressbar(tqdm):
    def update_to(self, gathered: int, total: int):
        self.total = total
        self.update(gathered - self.n)

pb = Progressbar(
            desc=f"downloading {filename}",
            unit="ib",
            unit_divisor=1 << 20, #MiB
            unit_scale=True,
        )

with pysftp.Connection(host, username, password) as sftp:
    sftp.get(filename, callback=pb.update_to)

Upvotes: 3

rekinyz
rekinyz

Reputation: 6602

import math, pysftp

with pysftp.Connection(host, username, password) as sftp:
    sftp.get(remotepath, localpath, callback=lambda x,y: progressbar(x,y))

def progressbar(x, y):
    ''' progressbar for the pysftp
    '''
    bar_len = 60
    filled_len = math.ceil(bar_len * x / float(y))
    percents = math.ceil(100.0 * x / float(y))
    bar = '=' * filled_len + '-' * (bar_len - filled_len)
    filesize = f'{math.ceil(y/1024):,} KB' if y > 1024 else f'{y} byte'
    sys.stdout.write(f'[{bar}] {percents}% {filesize}\r')
    sys.stdout.flush()

[============================================================] 100% 4,342 KB

Upvotes: 3

Manolis M. Tsangaris
Manolis M. Tsangaris

Reputation: 138

There exists a call back function in both get/put methods (see the official documentation)

get(remotepath, localpath=None, callback=None, preserve_mtime=False)

where callback can look like:

lambda x,y: print("{} transfered out of {}".format(x,y))

Upvotes: 11

PITFAR
PITFAR

Reputation: 111

You can create a function which prints the transfer every (lets say) 10 percent:

progressDict={}
progressEveryPercent=10

for i in range(0,101):
    if i%progressEveryPercent==0:
        progressDict[str(i)]=""

def printProgressDecimal(x,y):
    if int(100*(int(x)/int(y))) % progressEveryPercent ==0 and progressDict[str(int(100*(int(x)/int(y))))]=="":
        print("{}% ({} Transfered(B)/ {} Total File Size(B))".format(str("%.2f" %(100*(int(x)/int(y)))),x,y))
        progressDict[str(int(100*(int(x)/int(y))))]="1"

Then you can call that function in your get or put command like this:

sftp.get(eachFile,localpath=localDirectory+eachFile, callback=lambda x,y: printProgressDecimal(x,y))

A Sample Output Is:

0.00% (32768 Transfered(B)/ 1108907068 Total File Size(B))
10.00% (110919680 Transfered(B)/ 1108907068 Total File Size(B))
20.00% (221806592 Transfered(B)/ 1108907068 Total File Size(B))
30.00% (332693504 Transfered(B)/ 1108907068 Total File Size(B))
40.00% (443580416 Transfered(B)/ 1108907068 Total File Size(B))
50.00% (554467328 Transfered(B)/ 1108907068 Total File Size(B))
60.00% (665354240 Transfered(B)/ 1108907068 Total File Size(B))
70.00% (776241152 Transfered(B)/ 1108907068 Total File Size(B))
80.00% (887128064 Transfered(B)/ 1108907068 Total File Size(B))
90.00% (998047744 Transfered(B)/ 1108907068 Total File Size(B))
100.00% (1108907068 Transfered(B)/ 1108907068 Total File Size(B))

Upvotes: 9

msvalkon
msvalkon

Reputation: 12077

You can either write your own progressbar, which is fairly simple if you know anything about terminals. Or you can use progressbar. Here's a simple example from the documentation:

@example
def example1():
    widgets = ['Test: ', Percentage(), ' ', Bar(marker=RotatingMarker()),
               ' ', ETA(), ' ', FileTransferSpeed()]
    pbar = ProgressBar(widgets=widgets, maxval=10000000).start()
    for i in range(1000000):
        # do something
        pbar.update(10*i+1)
    pbar.finish()

In both cases, you will need the file transfer method to yield something during the transfer. If you can get it to yield the bytes it receives, it's very easy to create a progress bar either way.

Upvotes: -3

Related Questions