CODEWITHSUNDEEP
node.jsherokuurl-parsingnode-fetchurl-validation
patrickT
patrickT

Reputation: 11

My seemingly valid URLs parse locally but fail when deployed on heroku

This is my first question on stackoverflow, so please be gentle. I'm building a simple app using node.js that relies on API calls to an external geo-encoding service. My program runs locally, but fails when deployed to heroku. Here's the function that is failing:

async function geoEncode(restaurant = false, string = false) {
  let address;
  address = (string) ? string : require('querystring').escape(`${restaurant.street_address}, ${restaurant.city_state_zip}, ${restaurant.state}, ${restaurant.zip}`);
  let url = await geoApi.replace("SEARCH_STRING", address);
  let encodedURL = new URL(url);
  let init = {};
  let headers = {
    "async": true,
    "crossDomain": true,
    "method": "GET",
    "Host": "us1.locationiq.com",
  }
  init.headers = headers;
  try {
    let response = await fetch(encodedURL.href, init);
    let output = await response.json();
    return output;
  } catch(err){
    console.log("this log an error in geoEncode", err);
  }
}

I recognize that this code is currently a mess of unnecessary encoding and redundant headers. Side-effects of my failed debugging process. When run locally, the program sends a valid url to the service and returns responsive data. When deployed on heroku, 

let encodedURL = new URL(url);

throws 

TypeError [ERR_INVALID_URL]: Invalid URL: "https://us1.locationiq.com/v1/search.php?key ...

I have edited this url for my question to protect the API key, but when I take that string (minus the quotes) and run it through my browser it returns valid JSON.

Before deployment I wasn't doing any encoding on my urls at all - I just built a string and fed it to the node-fetch module's fetch method. It errored on Heroku, with the message "TypeError: Only absolute URLs are supported". This error message, I have since learned, means that node-fetch's request method is failing to parse either a protocol or a hostname. I have managed to determine that when deployed on heroku, it fails to parse both.

The obvious difference between my local environment and heroku's is that I have set my environmental variables through heroku's dashboard rather than through my .env file. Because I'm a little unclear about how that works most of my recent debugging has been trying to ensure that the string I'm feeding to the fetch API is in a consistent, uniform encoding. So I have pre-escaped my code, manually utf8 encoded it, run it through the URL module, and fed the output of that url-encoding into the function without touching it. It still works locally and fails remotely.

My basic stack is MongoDB, Express, Node, Bootstrap. Calls I make to my cloud MongoDB server work, as do calls to the API from which I source the data I geo-encode. Only these calls to my geo-encoding service are failing, so the environment variables I use for my MongoDB connection are functioning as expected.

Thanks for your help!

Edit 1:

In answer to a comment below (sorry about the formatting of my attempted comment-answer; I now know that markdown formatting is limited in comments), the errors I posted are logged from heroku using the command "heroku logs --tail". I have manually logged the URL I am trying to encode (what follows is an edited example):

URL to be encoded: "https://us1.locationiq.com/v1/search.php?key=KEY-REDACTED&format=json&q=101%20NICKERSON%20ST%2C%20SEATTLE%2C%20WA%2C%2098109-1654"

as well as the error heroku is reporting:

TypeError [ERR_INVALID_URL]: Invalid URL: "https://us1.locationiq.com/v1/search.php?key=KEY-REDACTED=json&q=101%20NICKERSON%20ST%2C%20SEATTLE%2C%20WA%2C%2098109-1654"

Edit 2:

I think I may have not provided enough error logging in my post. Here is a more complete error log. I hope this is helpful to someone who can in turn help me.

2019-11-14T23:35:09.646585+00:00 app[web.1]: (node:23) UnhandledPromiseRejectionWarning: TypeError [ERR_INVALID_URL]: Invalid URL: "https://us1.locationiq.com/v1/search.php?key=KEY_REDACTED&format=json&q=101%20NICKERSON%20ST%2C%20SEATTLE%2C%20WA%2C%2098109-1654"
2019-11-14T23:35:09.646591+00:00 app[web.1]: at onParseError (internal/url.js:243:9)
2019-11-14T23:35:09.646593+00:00 app[web.1]: at new URL (internal/url.js:319:5)
2019-11-14T23:35:09.646595+00:00 app[web.1]: at geoEncode (/app/server.js:308:20)
2019-11-14T23:35:09.646599+00:00 app[web.1]: at processTicksAndRejections (internal/process/task_queues.js:93:5)
2019-11-14T23:35:09.646602+00:00 app[web.1]: at async Timeout._onTimeout (/app/server.js:373:29)

Upvotes: 1

Views: 5505

Answers (1)

patrickT
patrickT

Reputation: 11

Resolved! The answer turns out to be predictably dull, but here it is:

When I entered the environmental variable referenced as geoAPI in the heroku dashboard, I entered it as a string with quotes. Heroku must already type environmental variables as strings, because the result was a string beginning and ending with "", which was causing the url module to fail to parse. I eventually discovered the problem when I console logged the output of querystring.parse(url) and received:

parsed querystring: [Object: null prototype] {
2019-11-19T21:54:59.724380+00:00 app[web.1]: '"https://us1.locationiq.com/v1/search.php?key': 'REDACTED',
2019-11-19T21:54:59.724382+00:00 app[web.1]: format: 'json',
2019-11-19T21:54:59.724385+00:00 app[web.1]: q: '101 NICKERSON ST, SEATTLE, WA, 98109-1654"' }

The extra quotes made everything up to the first '=' read as a key, apparently. And that's how I spotted the extra quotes. Removing the extra quotes resolved the problem. Glad it's over, but sad that it turned out to be just a formatting error in the heroku dashboard.

My small lesson from this: Don't put you heroku environmental variables in quotes. That's not how they work.

My big lesson from this: When something is mysterious, check your environment early. And if you check it, don't find anything, and stay stuck on the problem for a long time, check it again.

Upvotes: 0

Related Questions