CODEWITHSUNDEEP
wget
niCk cAMel
niCk cAMel

Reputation: 919

wget recursive returns BAD REQUEST

I've tried downloading files from different subdirs using wget

wget -r -nd --no-parent -e robots=off --auth-no-challenge --user=myusr --ask-password https://myexdomain.com/dir/subdir/

and it works without any problems.

When I however, make the wget call to one directory-level up

wget -r -nd --no-parent -e robots=off --auth-no-challenge --user=myusr --ask-password https://myexdomain.com/dir/

wget returns 400: BAD REQUEST

Any ideas why?

Upvotes: 1

Views: 2445

Answers (1)

Mekki MacAulay
Mekki MacAulay

Reputation: 1727

HTTP error 400 is fairly generic and usually related to the formatting of the client request. Without you providing the actual websites in question for us to test, it's a bit difficult to guess. However, here are a few guesses:

  • Does it work fine on plain http but not on https? If so, it may be related to a security policy.

  • Are you using a firewall, router, or resident antivirus that inspects packets? If so, it may be changing the packet structure rendering it invalid. If you're on a bigger network, is there such a device between you and the host in question (a tracert would help you figure that out).

  • Do you have access to this host server? If so, can you create new directories and test the exact conditions under which it fails? Is it always at the one-directory-below-root level?

  • You're excluding robots, which can be fine sometimes, but can also trigger different responses from servers depending on context. Have you tested without the robots exclusion?

  • Have you tried with a different user-agent? Some servers respond differently to wget user-agents. Can you access these files in a web-browser? Using curl? Or with a different user-agent specified with --user-agent in your wget command?

If none of this works, let us know the results of additional testing.

Upvotes: 1

Related Questions