CODEWITHSUNDEEP
httpswget
benck
benck

Reputation: 2052

my https website can't download by WGET command

I can browse the page by browser, but I can't download the html page by wget. https://money.benck.tw

When I use wget, it can't even connect to the website:

--2011-10-12 05:30:24--  https://money.benck.tw/
Resolving money.benck.tw... 97.107.135.68
Connecting to money.benck.tw|97.107.135.68|:443... failed: Connection timed out.
Retrying.

--2011-10-12 05:33:35--  (try: 2)  https://money.benck.tw/
Connecting to money.benck.tw|97.107.135.68|:443...

However, I can download the other https website like: https://ajax.googleapis.com/ajax/libs/jquery/1/jquery.min.js It's very weird.

Upvotes: 0

Views: 7533

Answers (3)

Daniel
Daniel

Reputation: 8677

This is because of this page is probably scraped by wget too often. You need to modify headers, especially useragent.

Examples from other website:

--no-check-certificate does not hepls

 wget --no-check-certificate "https://www.money.pl/pieniadze/depozyty/walutowearch/1921-02-05,2021-02-05,LIBORCHF3M,strona,1.html"                                                                  --2021-02-05 17:05:34--  https://www.money.pl/pieniadze/depozyty/walutowearch/1921-02-05,2021-02-05,LIBORCHF3M,strona,1.html
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving www.money.pl (www.money.pl)... 212.77.101.20
Connecting to www.money.pl (www.money.pl)|212.77.101.20|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2021-02-05 17:05:34 ERROR 403: Forbidden.

but other tool to download sendign other headers works

 http -h "https://www.money.pl/pieniadze/depozyty/walutowearch/1921-02-05,2021-02-05,LIBORCHF3M,strona,1.html"  
HTTP/1.1 200 OK
Cache-control: max-age=60, public,stale-while-revalidate=5
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 20756
Content-Security-Policy: upgrade-insecure-requests;
Content-Type: text/html; charset=iso-8859-2
Date: Fri, 05 Feb 2021 16:04:16 GMT
Link: <https://money.wp.pl/dGxwOTV0SyYZFTlneUtGM1pNbSY9EkhlJ1V1dglvOxgnKBALCW87GCcoEAsJbzsYJygQCwlvOxgnKBALCW87GCcoEAsJbzsYJygQCwlvOxgnKBALCW87GCcoEAsJbzsYJygQCwlvOxgnKBALCW87GCcoEAsJbzsYJygQCwlvOxgnKBALCW87GCcobXh0RUZ9WlgoNTAeDjRHBTlpZxYWIhMeKydrAld1TER2ciZYECoUSjgjIR4JKBYSNnomXEF1TUUJJD9VCi4ZEzUxcwJRdT4TKiQ5Sh0zAVJ9YWR2EyYUAjs7IVUFNRsfamZjAiJ2QUV-eWYCSXdNUn1hZHNWd0pGYmRkHVRyXUV6ZhV8LQU3JQwcEAMpYkpCfRclRBYoFhZqZmMCJ3ZWHzs5OhY0EDkoLjA0VFl1XgQ_PTgNKRMbQgIuB0lCIRQEOzUiWQB6XhYrIgVcCzMLSn9lZhYHJBkDKjM5Qh16DxYjISJJRjo=>;rel="preload";as="script";
Server: nginx
Set-Cookie: mny_ver2=v8c;Domain=.money.pl;Path=/;Max-Age=2592000;
Vary: Accept-Encoding

Upvotes: 0

Francisco Yu
Francisco Yu

Reputation: 1

I'm experiments the same issue, I trying to download files from an external site like https://downloads.wordpress.org/plugin/easy-wp-smtp.zip and I wget using --no-check-certificate stills not working.... It's freezing in this line:

Connecting to downloads.wordpress.org (downloads.wordpress.org)|198.143.164.250|:443...

Anyone have the same issue?

No IP tables configured and rules. When I do this on other server on the same networks works fine. This only happens on this server specialy.

Regards, Francisco Yu

Upvotes: 0

serk
serk

Reputation: 4399

For this website you have to use the --no-check-certificate command

wget --no-check-certificate https://money.benck.tw

Upvotes: 2

Related Questions