CODEWITHSUNDEEP
google-apps-scripturlfetch
jason
jason

Reputation: 4439

Google Apps Script: 403 error for UrlFetchApp

I'm using google app script

var rss = "https://www.sec.gov/cgi-bin/browse-edgar?action=getcurrent&CIK=&type=8-k&company=&dateb=&owner=exclude&start=0&count=100&output=atom"
var r = UrlFetchApp.fetch(rss).getContentText()

but once a while, I get a failed execution and this is the response. It is about half the time.

Exception: Request failed for https://www.sec.gov returned code 403. Truncated server response: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w... (use muteHttpExceptions option to examine full response)

Not sure why it is happening and how to fix it.

Upvotes: 2

Views: 4486

Answers (2)

crew96
crew96

Reputation: 21

This can occur when your request rate exceeds the "fair use" definition, which is currently 10 requests per second.

https://www.sec.gov/os/accessing-edgar-data

You can confirm whether this is the case by examining the response body.

Upvotes: 2

Tanaike
Tanaike

Reputation: 201428

Although, unfortunately, I cannot replicate your situation, from but once a while, I get a failed execution and this is the response., for example, how about retrying the request as follows?

Sample script:

var res = null;
var maxRetries = 5;
var rss = "https://www.sec.gov/cgi-bin/browse-edgar?action=getcurrent&CIK=&type=8-k&company=&dateb=&owner=exclude&start=0&count=100&output=atom";
for (var i = 0; i < maxRetries; i++) {
  res = UrlFetchApp.fetch(rss, {muteHttpExceptions: true});
  if (res.getResponseCode() == 200) break;
  res = null;
  Utilities.sleep(5000);
}
if (!res) throw new Error("Values cannot be retrieved.");

// do something.
var value = res.getContentText();
  • In this sample script, when the status code is not 200, after 5 seconds, the request is retried. And the max number of retries is 5 times.
  • Please modify maxRetries and Utilities.sleep(5000) for your actual situation.

Reference:

Upvotes: 5

Related Questions