CODEWITHSUNDEEP
linuxcronwget
Ali
Ali

Reputation: 267059

Does WGET timeout?

I'm running a PHP script via cron using Wget, with the following command:

wget -O - -q -t 1 http://www.example.com/cron/run

The script will take a maximum of 5-6 minutes to do its processing. Will WGet wait for it and give it all the time it needs, or will it time out?

Upvotes: 107

Views: 197284

Answers (5)

Earl Ruby
Earl Ruby

Reputation: 1570

None of the wget timeout values have anything to do with how long it takes to download a file.

If the PHP script that you're triggering sits there idle for 5 minutes and returns no data, wget's --read-timeout will trigger if it's set to less than the time it takes to execute the script.

If you are actually downloading a file, or if the PHP script sends some data back, like a ... progress indicator, then the read timeout won't be triggered as long as the script is doing something.

wget --help tells you:

  -T,  --timeout=SECONDS           set all timeout values to SECONDS
       --dns-timeout=SECS          set the DNS lookup timeout to SECS
       --connect-timeout=SECS      set the connect timeout to SECS
       --read-timeout=SECS         set the read timeout to SECS

So if you use --timeout=10 it sets the timeouts for DNS lookup, connecting, and reading bytes to 10s.

When downloading files you can set the timeout value pretty low and as long as you have good connectivity to the site you're connecting to you can still download a large file in 5 minutes with a 10s timeout. If you have a temporary connection failure to the site or DNS, the transfer will time out after 10s and then retry (if --tries aka -t is > 1).

For example, here I am downloading a file from NVIDIA that takes 4 minutes to download, and I have wget's timeout values set to 10s:

$ time wget --timeout=10 --tries=1 https://developer.download.nvidia.com/compute/cuda/11.2.2/local_installers/cuda_11.2.2_460.32.03_linux.run
--2021-07-02 16:39:21--  https://developer.download.nvidia.com/compute/cuda/11.2.2/local_installers/cuda_11.2.2_460.32.03_linux.run
Resolving developer.download.nvidia.com (developer.download.nvidia.com)... 152.195.19.142
Connecting to developer.download.nvidia.com (developer.download.nvidia.com)|152.195.19.142|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3057439068 (2.8G) [application/octet-stream]
Saving to: ‘cuda_11.2.2_460.32.03_linux.run.1’

cuda_11.2.2_460.32.03_linux.run.1        100%[==================================================================================>]   2.85G  12.5MB/s    in 4m 0s

2021-07-02 16:43:21 (12.1 MB/s) - ‘cuda_11.2.2_460.32.03_linux.run.1’ saved [3057439068/3057439068]


real    4m0.202s
user    0m5.180s
sys 0m16.253s

4m to download, timeout is 10s, everything works just fine.

In general, timing out DNS, connections, and reads using a low value is a good idea. If you leave it at the default value of 900s you'll be waiting 15m every time there's a hiccup in DNS or your Internet connectivity.

Upvotes: 3

Dean Rather
Dean Rather

Reputation: 32374

Prior to version 1.14, wget timeout arguments were not adhered to if downloading over https due to a bug.

Upvotes: 8

Pascal MARTIN
Pascal MARTIN

Reputation: 400932

According to the man page of wget, there are a couple of options related to timeouts -- and there is a default read timeout of 900s -- so I say that, yes, it could timeout.


Here are the options in question :

-T seconds
--timeout=seconds

Set the network timeout to seconds seconds. This is equivalent to specifying --dns-timeout, --connect-timeout, and --read-timeout, all at the same time.


And for those three options :

--dns-timeout=seconds

Set the DNS lookup timeout to seconds seconds.
DNS lookups that don't complete within the specified time will fail.
By default, there is no timeout on DNS lookups, other than that implemented by system libraries.

--connect-timeout=seconds

Set the connect timeout to seconds seconds.
TCP connections that take longer to establish will be aborted.
By default, there is no connect timeout, other than that implemented by system libraries.

--read-timeout=seconds

Set the read (and write) timeout to seconds seconds.
The "time" of this timeout refers to idle time: if, at any point in the download, no data is received for more than the specified number of seconds, reading fails and the download is restarted.
This option does not directly affect the duration of the entire download.


I suppose using something like 

wget -O - -q -t 1 --timeout=600 http://www.example.com/cron/run

should make sure there is no timeout before longer than the duration of your script.

(Yeah, that's probably the most brutal solution possible ^^ )

Upvotes: 180

hIpPy
hIpPy

Reputation: 5085

The default timeout is 900 second. You can specify different timeout.

-T seconds
--timeout=seconds

The default is to retry 20 times. You can specify different tries.

-t number
--tries=number

link: wget man document

Upvotes: 46

Marco Demaio
Marco Demaio

Reputation: 34407

Since in your question you said it's a PHP script, maybe the best solution could be to simply add in your script:

ignore_user_abort(TRUE);

In this way even if wget terminates, the PHP script goes on being processed at least until it does not exceeds max_execution_time limit (ini directive: 30 seconds by default).

As per wget anyay you should not change its timeout, according to the UNIX manual the default wget timeout is 900 seconds (15 minutes), whis is much larger that the 5-6 minutes you need.

Upvotes: 7

Related Questions