CODEWITHSUNDEEP
shellcurl
Jabda
Jabda

Reputation: 1792

Curl in Bash, Redirect STDERR to file except for progress stderr

My shell script uploads files to a server. I'd like the stdout and stderr to write to both a file and the console. BUT I don't want the progress bar/percentage which is stderr to go to file. I only want curl errors to write to file.

Initially I had this

curl ...  2>> "$log"

This wrote nice neat 1 or more lines of the download to the log file, but nothing to console.

I then changed it to 

curl ... 3>&1 1>&2 2>&3 | tee -a "$log"

This wrote to both console and file, yay! except it wrote the whole progress for each percentage to the file, making the log file very large and tedious to read.

How can I view the progress in the console, but only write the last part of the output to file?

I want this

 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                Dload  Upload   Total   Spent    Left  Speed
  106  1166    0     0  106  1166      0   2514 --:--:-- --:--:-- --:--:--  2 106  1166    0     0  106  1166      0    795  0:00      :01  0:00:01 --:--:--     0 106  1166    0     0  106  1166      0    660  0:00:01  0:00:01 --:--:--     0

This is what I get with the second curl redirect

 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                             Dload  Upload   Total   Spent    Left  Speed



 0  9.8G    0     0    0 16384      0  24764   4d 22h --:--:--   4d 22h 24764
  0  9.8G    0     0    0 3616k      0  4098k  0:41:54 --:--:--  0:41:54 15.9M
  0  9.8G    0     0    0 24.2M      0  12.9M  0:12:59  0:00:01  0:12:58 19.8M
  0  9.8G    0     0    0 50.4M      0  17.5M  0:09:34  0:00:02  0:09:32 22.7M
  0  9.8G    0     0    0 79.8M      0  20.5M  0:08:09  0:00:03  0:08:06 24.7M
  1  9.8G    0     0    1  101M      0  20.7M  0:08:04  0:00:04  0:08:00 24.0M
  1  9.8G    0     0    1  129M      0  21.9M  0:07:37  0:00:05  0:07:32 25.1M
  1  9.8G    0     0    1  150M      0  21.8M  0:07:41  0:00:06  0:07:35 25.1M
  1  9.8G    0     0    1  169M      0  21.5M  0:07:47  0:00:07  0:07:40 23.8M
  1  9.8G    0     0    1  195M      0  21.9M  0:07:38  0:00:08  0:07:30 23.0M
  2  9.8G    0     0    2  219M      0  22.1M  0:07:33  0:00:09  0:07:24 23.5M
  2  9.8G    0     0    2  243M      0  22.4M  0:07:29  0:00:10  0:07:19 22.9M
  2  9.8G    0     0    2  273M      0  22.9M  0:07:17  0:00:11  0:07:06 24.6M
..
.. hundreds of lines...
..
99  9.8G    0     0   99 9982M      0  24.8M  0:06:45  0:06:41  0:00:04 24.5M
 99  9.8G    0     0   99  9.7G      0  24.8M  0:06:44  0:06:42  0:00:02 24.9M
 99  9.8G    0     0   99  9.8G      0  24.8M  0:06:44  0:06:43  0:00:01 26.0M
100  9.8G    0     0  100  9.8G      0  24.8M  0:06:44  0:06:44 --:--:-- 25.8M

Edit: What I dont understand is according to http://www.tldp.org/LDP/abs/html/io-redirection.html

2>filename
  # Redirect stderr to file "filename."

but if I use that I don't get every single stderr progress line in the file. If I try any of the other solutions every stderr progress line is redirected to file

Upvotes: 0

Views: 1887

Answers (3)

WesternGun
WesternGun

Reputation: 12728

use curl -s to suppress progress. It would not show anywhere. (maybe not your case)

use curl -o to redirect response body to file. The file has to be created by you, if it's another machine where you SSH to.

use curl --write-out FORMAT to organize the output in stdout. You can then redirect to wherever you want with >.

user 2>&1 to copy file descriptor, so 2 is redirected to where the &1 is redirected, could be file or stdout.

I don't see a way to just get last line of progress. Why do you need it?

Upvotes: 0

hagope
hagope

Reputation: 5531

Let's say you have 10 huge .txt files test1.txt ... test10.txt

Here's how you would upload them with a single cURL command and log the results without logging the progress meter, the trick is to use the --write-out or -w option, from the man file, these are all the relevent fields for uploading to FTP:

curl -T "test[1-10:1].txt" -u user:password sftp://example.com:22/home/user/ -# -k -w "%{url_effective}\t%{ftp_entry_path}\t%{http_code}\t%{http_connect}\t%{local_ip}\t%{local_port}\t%{num_connects}\t%{num_redirects}\t%{redirect_url}\t%{remote_ip}\t%{remote_port}\t%{size_download}\t%{size_header}\t%{size_request}\t%{size_upload}\t%{speed_download}\t%{speed_upload}\t%{ssl_verify_result}\t%{time_appconnect}\t%{time_connect}\t%{time_namelookup}\t%{time_pretransfer}\t%{time_redirect}\t%{time_starttransfer}\t%{time_total}\n" >> log.txt

For your log.txt file you may want to pre-pend the column headers first:

echo -e "url_effective\tftp_entry_path\thttp_code\thttp_connect\tlocal_ip\tlocal_port\tnum_connects\tnum_redirects\tredirect_url\tremote_ip\tremote_port\tsize_download\tsize_header\tsize_request\tsize_upload\tspeed_download\tspeed_upload\tssl_verify_result\ttime_appconnect\ttime_connect\ttime_namelookup\ttime_pretransfer\ttime_redirect\ttime_starttransfer\ttime_total" > log.txt The -# makes the progress bar a bit neater like: ######################################################################## 100.0% ######################################################################## 100.0%

...and the curl -T "test[1-10:1].txt" piece lets you specify a range of files to upload.

Upvotes: 0

user4427511
user4427511

Reputation:

Just as a video is a series of frames, an updating percentage in the console is a series of lines. What is in the file is the true output. The difference is a carriage return in a text file puts the following text below, whereas in the console it overwrites the current line.

If you want to see the updating percentage in the console, but not the file, you could use something like this:

curl |& tee >(sed '1b;$!d' > log)

Or:

curl |& tee /dev/tty | sed '1b;$!d' > log

Upvotes: 1

Related Questions