Is there any way to use UrlFetchApp.fetch to get Cookies in Google App Script?

Question

i am looking for a way to get Cookies using UrlFetchApp. The code I'm using from someone on the forum:

function tele() {

  var payload = { 'user_login': 'user_login', 'user_pass': 'user_pass' };
  //replace with values on your login page "name=user_name" and "name=password"
  //if your username contains @ send it directly or use %40 instead
  var opt = {
    'payload': payload,
    'method': 'get',
    'authority':'trader.myforexfunds.app',
    "validateHttpsCertificates":true,
    "followRedirects": true,
    "testcookie": 1
  };
  var response = UrlFetchApp.fetch("https://trader.myforexfunds.app/?login=20091997&interval=h", opt);
  
  //inspect the right link via Chrome inspect of your login page
  console.log(response.getAllHeaders());
  console.log(response.getResponseCode());
  if (response.getResponseCode() == 302) {
    // Incorrect user/pass combo
  } else if (response.getResponseCode() == 200) {
    // Logged-in
    console.log("Đã đăng nhập");
    var headers = response.getAllHeaders();
    console.log(headers);
    var cookies = headers['Set-Cookie'];
    // Extract the cookies from the login response
    var cookieParts = [];
    for (var i = 0; i < cookies.length; i++) {
      var arr = cookies[i].split('; ');
      cookieParts.push(arr[0]);
    }
    // Create a new cookie to send with subsequent requests
    var newCookie = cookieParts.join('; ');
    console.log("Cookies mới là :"+newCookie);
  };


  opt2 = {
    "method": "get",
    "headers": {
      "Cookie": newCookie
      
    }
  };
  var url = "https://trader.myforexfunds.app/api/data/trade/20091997/71264346";
  response2 = UrlFetchApp.fetch(url, opt2);
  var resp1 = response2.getContentText();
  //var csvContent = parseCsvResponse(response2.getContentText());
  Logger.log(resp1);
  //Logger.log(csvContent);
  // clear everything in the sheet
  //var sheet = SpreadsheetApp.getActiveSheet();
  //sheet.clearContents().clearFormats();

  // set the values in the sheet (as efficiently as we know how)
  //sheet.getRange(1, 1, csvContent.length /* rows */, csvContent[0].length /* columns */).setValues(csvContent);
}

+ I'm having a problem : Cookies cannot be obtained

Console Return :

  { 'Content-Type': 'text/html; charset=utf-8',
  'Cache-Control': 'no-cache,no-store',
  Expires: 'Thu, 01 Jan 1970 00:00:00 GMT',
  'Set-Cookie': 
   [ '.AspNetCore.Cookies=; expires=Thu, 01 Jan 1970 00:00:00 GMT; path=/; samesite=lax; httponly',
     '.AspNetCore.Antiforgery.DvnwCO4RNgs=CfDJ8IT31SjLt0NIjQqQscrPrXVqUn_4QOPIbvLFyNztSQBxfrkhs4wQjPbnpSybF11CUCgRzQEGDAJk3BUr7GLSVpN_CPDxcOH3-uemnwJTUEO8rQlkVbYMFKeARM8Jtf4382MpQuZb9ZJtBlo6KR_JUyU; path=/; samesite=strict; httponly' ],
  Connection: 'keep-alive',
  'Transfer-Encoding': 'chunked',
  nel: '{"success_fraction":0,"report_to":"cf-nel","max_age":604800}',
  'cf-cache-status': 'DYNAMIC',
  Date: 'Wed, 10 Aug 2022 09:01:28 GMT',
  'X-Frame-Options': 'SAMEORIGIN',
  Vary: 'Accept-Encoding',
  'cf-ray': '73878c0eac549c22-IAD',
  Server: 'cloudflare',
  'Content-Encoding': 'br',
  'Report-To': '{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=g%2BlJNlosGcLvwGADcq7dhPUji5jREUZTV%2BQfPq8OyD0M%2F%2BLnJ7YwrUiVfj4r3F74PrLk3Axhj0YXjTUc7goXJ1R4ndHZNEMwkEusyD12By8%2Bdg2F8hxokIMZtJw7KGoFDVSr9rQLdd27"}],"group":"cf-nel","max_age":604800}',
  'expect-ct': 'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"',
  Pragma: 'no-cache' }

But the value of the entry .AspNetCore.Cookies= "Empty" so it is not possible to get the complete Cookies,Hope you guys can help me adjust so I can Fix it. Thanks You.

Answer 1

Apps Script is returning the right Cookies. The problem is that you're not actually authenticating. Try to fetch the page without sending the payload and you will get pretty much the same headers.

Furthermore, If you try to manually log in to https://trader.myforexfunds.app/?login=20091997&interval=h and inspect the traffic you will see that the form data is different from what you have in your payload:

As you can see, your script uses user_login and user_pass, but the right values are username and password, in addition, there's a __RequestVerificationToken field that also gets sent. I'm not familiar with that but in this question it's explained that it's an anti-forgery token and needs to be included in the POST data along with its corresponding Cookie or your request will be rejected.

The token is in a hidden field within the same page so as explained in the question I linked above, you need to fetch the page, get the AspNetCore.Antiforgery... Cookie, get the __RequestVerificationToken value in the hidden field, then send the POST request including this Cookie and the payload with the token. Here's also another nice answer that explains how to handle this token.

With all this in mind your payload and headers should look more like this:

var payload = {
    'username': your_username,
    'password': your_password,
     '__RequestVerificationToken': the_token
};

var opt = {
    'payload': payload,
    'method': 'POST',
    'authority':'trader.myforexfunds.app',
    "validateHttpsCertificates":true,
    "followRedirects": true,
    "cookie": the_antiforgery_cookie
};

Aside from all this your script takes a 302 error code as a failed attempt, but the website doesn't return 302 error codes when entering the wrong credentials. This tells us that this script was not designed for this website, or it's for an older version, or maybe you just pasted your own values on a generic script. I'm not sure what forum you got it from but it needs more work to tailor it to this website.

Either way my recommendation is to rewrite the script while taking into account what I mentioned above. Unfortunately there is no way for me to test this to see if there are any additional hurdles because the user registration is behind a paywall, and you need valid credentials to go any further. I hope this helps.