CODEWITHSUNDEEP
curlwget
binit
binit

Reputation: 476

Wget vs Curl: getting site output with forbidden 403 error

I am using the following command:

wget --no-check-certificate "https://www.googleapis.com/youtube/v3/search?alt=json&q=hello&part=snippet"

I get 403 Forbidden error (it is the expected output), but there is also an accompanying server response as the reason which I am not able to get. I tried with various options - ignoring content length, giving a private PEM certificate, setting user-agent, etc. but still not able to get that output.

Using curl however, the output comes as expected. curl "https://www.googleapis.com/youtube/v3/search?alt=json&q=hello&part=snippet"

Wget Output:

--2013-06-07 18:37:13--  https://www.googleapis.com/youtube/v3/search?alt=json&q=hello&part=snippet
Resolving www.googleapis.com (www.googleapis.com)... 74.125.135.95, 2404:6800:4001:c01::5f
Connecting to www.googleapis.com (www.googleapis.com)|74.125.135.95|:443... connected.
WARNING: The certificate of `www.googleapis.com' is not trusted.
WARNING: The certificate of `www.googleapis.com' hasn't got a known issuer.
HTTP request sent, awaiting response... 403 Forbidden
2013-06-07 18:37:14 ERROR 403: Forbidden.

Curl Output:

{
 "error": {
  "errors": [
   {
    "domain": "usageLimits",
    "reason": "dailyLimitExceededUnreg",
    "message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup.",
    "extendedHelp": "https://code.google.com/apis/console"
   }
  ],
  "code": 403,
  "message": "Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup."
 }
}

The problem is that many of my machines have only wget installed, so this is being an annoying issue to me. I'm almost sure I missed some option which makes wget show the output. If you have any ideas, I'd be happy to try it out. Thanks.

Upvotes: 2

Views: 6994

Answers (1)

Ram Rajamony
Ram Rajamony

Reputation: 1723

Unfortunately, you are out of luck using wget.

The wget program only handles HTTP protocol version 1.0. In HTTP 1.1, a response can be sent using a chunked encoding. This means that either agent can send data in chunks, preceding each chunk with the size of the chunk being sent. This makes sense when sending large amounts of data OR with data whose size is not known ahead of time. Among other things, chunked encoding lets you move to different parts of the resource being transferred.

Whats happening in your case is that when you try to talk to Youtube over HTTP/1.0, the server responds saying that you're out of luck. When you talk to it over HTTP/1.1, it provides you a chunked response that contains the JSON you see. curl handles HTTP/1.1 and therefore handles the chunked transfer encoding supplied by Youtube. This Youtube clip gives you a quick peek into the chunked encoding.

Finally, the following is somewhat off-topic: Chunked transfer encodings and ensuing range requests and response are fairly involved. Kit Cambridge did a really nice presentation on them

Upvotes: 2

Related Questions