CODEWITHSUNDEEP
haskell
reallymemorable
reallymemorable

Reputation: 1024

Hitting weird character encoding issue with Haskell's req web scraping library

I can't make sense of an error occurring within this code:

import Network.URI.Encode as EN

downloadFile :: (MonadHttp m) => String -> m ()
downloadFile url = do

    traceShowM (T.pack url)

    -- Pull them into memory
    contents <- req GET (https (T.pack $ EN.encode url)) NoReqBody bsResponse mempty

    -- debug read file wrote file etc

    -- Get the filename itself
    let fileName = head $ reverse $ T.splitOn "/" (T.pack url)

This is the output of the main function (eatChunks):


*Main UsCensusDataLodesScraper> eatChunks
"https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2003.csv.gz"
"https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2002.csv.gz"
"https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2004.csv.gz"
"https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2006.csv.gz"
"https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2008.csv.gz"
"https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2009.csv.gz"
"https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2007.csv.gz"
"https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2010.csv.gz"
"https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2011.csv.gz"
*** Exception: VanillaHttpException (HttpExceptionRequest Request {
  host                 = "https%253A%252F%252Flehd.ces.census.gov%252Fdata%252Flodes%252FLODES7%252Fak%252Fod%252Fak_od_aux_JT00_2002.csv.gz"
  port                 = 443
  secure               = True
  requestHeaders       = []
  path                 = ""
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 (ConnectionFailure Network.Socket.getAddrInfo (called with preferred socket type/protocol: AddrInfo {addrFlags = [AI_ADDRCONFIG], addrFamily = AF_UNSPEC, addrSocketType = Stream, addrProtocol = 6, addrAddress = <assumed to be undefined>, addrCanonName = <assumed to be undefined>}, host name: Just "https%253A%252F%252Flehd.ces.census.gov%252Fdata%252Flodes%252FLODES7%252Fak%252Fod%252Fak_od_aux_JT00_2002.csv.gz", service name: Just "443"): does not exist (nodename nor servname provided, or not known)))

Upvotes: 1

Views: 95

Answers (2)

Jack Henahan
Jack Henahan

Reputation: 1386

You're not constructing the Url correctly. From the docs at

http://hackage.haskell.org/package/req-2.0.1/docs/Network-HTTP-Req.html#g:5

we see that a Url can be constructed in a few ways.

  1. By hand:
https "lehd.ces.census.gov"
      /~ "data"
      /~ "lodes"
      /~ "LODES7"
      /~ "ak"
      /~ "od"
      /~ "ak_od_aux_JT00_2003.csv.gz"
  1. By parsing:
-- Don't actually use fromJust, this is just an example
fst . fromJust $ parseUrlHttps "https://lehd.ces.census.gov/data/lodes/LODES7/ak/od/ak_od_aux_JT00_2003.csv.gz"

In particular, notice what https actually does: given a hostname, it constructs a Url with the HTTPS scheme, that hostname, and an empty path. https someUrlString will construct a Url like

https://<the fully url-encoded version of someUrlString>

which is not what you want.

Upvotes: 1

Daniel Wagner
Daniel Wagner

Reputation: 153247

You almost certainly do not want to call EN.encode. Probably just delete that.

Upvotes: 0

Related Questions