CODEWITHSUNDEEP
google-apps-scriptgoogle-sheetsweb-scrapingurlfetch
Ignacio Javier Kairuz Eguia
Ignacio Javier Kairuz Eguia

Reputation: 11

Timeout Error - DHL API to Google Sheets - UrlFetchApp

In Python I use as headers the "Request Headers" that are in the request captured using the browser's developer options and it works fine.

I tried the same with Apps Script, but UrlFetchApp retrieves Timeout exception:

function WS() {
  var myHeaders = {
  'accept': '*/*',
  'accept-encoding': 'gzip, deflate, br',
  'accept-language': 'en-US,en;q=0.9,es;q=0.8,pt;q=0.7',
  'cookie': '', //the cookies that appear here in my browser
  'referer': 'https://www.dhl.com/global-en/home/tracking/tracking-express.html?submit=1&tracking-id=4045339815',
  'sec-ch-ua': '"Microsoft Edge";v="105", "Not)A;Brand";v="8", "Chromium";v="105"',
  'sec-ch-ua-mobile': '?0',
  'sec-ch-ua-platform': '"Windows"',
  'sec-fetch-dest': 'empty',
  'sec-fetch-mode': 'cors',
  'sec-fetch-site': 'same-origin',
  'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.53',
  'x-sec-clge-req-type': 'ajax',
  };
  var options = {
    'method': 'GET',
    'headers': myHeaders,
}
  var response = UrlFetchApp.fetch("https://www.dhl.com/utapi?trackingNumber=4045339815&language=en&source=tt",options);
  Logger.log(response.getContentText())
};

I would appreciate any ideas / hint.

EDIT :

Website to catch cookies : https://www.dhl.com/global-en/home/tracking/tracking-express.html?submit=1&tracking-id=4045339815

Upvotes: 0

Views: 478

Answers (1)

Daniel
Daniel

Reputation: 3725

I think the problem is most likely the user-agent header. Apps Script's URL Fetch Service uses Google's servers to send the request instead of your browser. As a result, Apps Script forces its own user agent that looks like this:

"User-Agent": "Mozilla/5.0 (compatible; Google-Apps-Script; beanserver; +https://script.google.com; id: ...)"

On the other hand, Python sends the headers exactly as you specified them. You can test this yourself by sending your requests to a test server like https://httpbin.org/headers. The only difference between the Python and Apps Script requests is the user-agent header.

It doesn't look like there's a way to bypass this. There's a request in Google's issue tracker here to allow customization of the user agent but it's been open since 2013 so it doesn't seem like something they want to do, maybe for transparency reasons or something similar.

The reason why this header would be a problem is because DHL doesn't want you to use their user-facing endpoints to request information with scripts, though you probably already know this since you're trying to replicate the browser's headers and cookies. Trying to access the endpoint without the right headers just results in this message:

enter image description here

My guess is that DHL has blacklisted the Apps Script user agent, hence the timeout. If you want to use Apps Script you probably will have to go to https://developer.dhl and set up a developer account to get your own API key. If you want to keep using your current method then you'll have to stick to Python or anything else that won't change your headers.

Edit:

Here's a quick Python sample that seems to support the theory:

import requests
#Chrome user agent, this works
useragent = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36 Edg/105.0.1343.53'

#No user agent, this also works
#useragent = ''

#Fake user agent, this still works
#useragent = 'Mozilla/5.0 (compatible; Googlu-Opps-Script)'

#Apps Script user agent, this just hangs
#useragent = 'Mozilla/5.0 (compatible; Google-Apps-Script)'

headers= {
  'accept': '*/*',
  'accept-encoding': 'gzip, deflate, br',
  'accept-language': 'en-US,en;q=0.9,es;q=0.8,pt;q=0.7',
  'cookie': 'your-cookie',
  'referer': 'https://www.dhl.com/global-en/home/tracking/tracking-express.html?submit=1&tracking-id=4045339815',
  'sec-ch-ua': '"Microsoft Edge";v="105", "Not)A;Brand";v="8", "Chromium";v="105"',
  'sec-ch-ua-mobile': '?0',
  'sec-ch-ua-platform': '"Windows"',
  'sec-fetch-dest': 'empty',
  'sec-fetch-mode': 'cors',
  'sec-fetch-site': 'same-origin',
  'user-agent': useragent,
  'x-sec-clge-req-type': 'ajax'}

url="https://www.dhl.com/utapi?trackingNumber=4045339815&language=en&source=tt"

result = requests.get(url, headers=headers)

print(result.content.decode())

Based on my testing in Python, even a blank or fake user agent will work, but one that has Google-Apps-Script will just keep hanging. Even changing a single letter to Google-Opps-Script or something similar will make it work.

Upvotes: 0

Related Questions