Ryan Kohl
Ryan Kohl

Reputation: 63

Occasional null results with clojure's http.async.client and google

I have a result set in json with city names, and I'd like to get the lat-long for each. The following function works to an extent:

(:require [http.async.client :as client])

(defn get-geo-fact [row]
  (let [ 
    n (string/replace (:cityname row) " "  "+") 
    url (str "http://maps.googleapis.com/maps/api/geocode/json?address=" 
              n "&sensor=false")
    resp (client/GET url) ]
  (client/await resp)
  (make-geo-fact row (json/read-json (client/string resp))) ))

That last call to make-geo-fact just returns an RDF rendering of the city coordinates. The problem I'm running into is that a run (of about 40 calls to this function) returns a few (3-5 lat-long pairs) null results for lat-longs. The cities that return null values differ from run to run - sometimes San Jose gets coordinates, and sometimes it doesn't.

I originally used slurp to grab the body of the url and got similarly-occasional nulls. I figured I wasn't waiting for the response properly, but switching to http.async.client doesn't seem to be doing the trick. Any ideas?

edit:

Here's the make-geo-fact function, which takes a "this team is located in this city" fact and a response from google maps, returning a vector of two triples that convey the latitude and longitude:

(defn make-geo-fact [row response]
  (let [ g (:location (:geometry (first (:results response))))
         lat (str "<" (:team row) ">" 
                  " <http://www.nhl.com/latitude> \"" (:lat g) 
                  "\"^^<http://www.w3.org/2001/XMLSchema#decimal> ." )
         lon (str "<" (:team row) ">" 
                  " <http://www.nhl.com/longitude> \"" (:lng g) 
                  "\"^^<http://www.w3.org/2001/XMLSchema#decimal> ." )  ]
        [lat lon] ))

And here's the function I call to kick the whole thing off:

(defn make-geo-facts []
  (let [ a (bounce team-city (build "files/team-city.nt")) 
         f "files/geo-facts.nt" ]
    (spit f (string/join "\n" (flatten (map get-geo-fact (:rows a)))))  
    f ))

Where the bounce function issues a SPARQL select query against an RDF model, which is instantiated with the build function.

edit 2

Here's a re-factor where make-geo-fact isn't needed:

  (defn get-geo-fact [row]
    (let [ n (string/replace (:cityname row) " "  "+") 
           url (str "http://maps.googleapis.com/maps/api/geocode/json?address=" n "&sensor=false")
           resp (client/GET url)  ]
      (-> (client/GET url)
           client/await
           client/string
           json/read-json
           :results
           first
           :geometry
           :location )))

    (defn make-geo-facts []
        (let [ a (bounce tc-query (build "files/team-city.nt")) 
               f "files/geo-facts.nt"
               *client* (client/create-client)]
          (try (spit f (string/join "\n" (flatten (map get-geo-fact (:rows a))))))
          (finally (client/close *client*)) ))

Upvotes: 1

Views: 217

Answers (2)

Ryan Kohl
Ryan Kohl

Reputation: 63

Turns out my code needed a little sleep. I added (Thread/sleep 1000) to my core function and now I don't get null results:

(defn get-geo-fact [row]
    (let [ n (string/replace (:cityname row) " "  "+") 
            url (str "http://maps.googleapis.com/maps/api/geocode/json?address=" n "&sensor=false")
            resp (client/GET url)  ]
            (Thread/sleep 1000) 
        (-> (client/GET url)
              client/await
              client/string
              json/read-json
              (make-geo-fact ,,, row ) )))

Upvotes: 1

Hubert
Hubert

Reputation: 2313

As you've said changing client implementations didn't make a difference. I double checked and create a test for development version of http.async.client. And always got responses with body.

Please provide make-geo-fact implementation.

Upvotes: 1

Related Questions