CODEWITHSUNDEEP
concurrencyclojurepromisecore.async
Joe
Joe

Reputation: 47659

Waiting on a collection of promises?

I have a seq of promises. I want to wait until they're all available, and then deref to get the whole vector.

My actual use case is using HTTPKit to make make a number of concurrent requests and then process them when all have returned.

The ordering of the collection is important, but could be faked and re-ordered after the event.

I'm currently doing something like this:

(defn parallel-api-query
  [queries]
  (let [qs (map http/get queries)
        bodies (map #(json/read-str (:body @%)) qs)]
    bodies))

All the queries are issued in the qs binding and then the map will block on each one in turn until its dereffed in the map.

Will this achieve the outcome I want?

Upvotes: 4

Views: 426

Answers (1)

Alex
Alex

Reputation: 13961

This implementation might appear to work at first, especially if you're testing the function in isolation by passing in a vector of query URLs, due to Clojure's use of chunked sequences and how they're handled by the map function. However, you shouldn't rely on that behavior, because you'll lose parallelism if the calling code starts passing in some non-chunked sequence (such as a list, or another lazy sequence that was computed from who knows what input).

Keep in mind that map is lazy, so binding qs as (map http/get queries) will yield a lazy sequence, that will not submit the requests until it's forced to do so. The error is compounded by the fact that (map #(json/read-str (:body @%)) qs) returns another lazy sequence that will process the (lazy) qs sequence, submitting API calls and then blocking on them one at a time as the caller of your function requests items from the returned sequence. In the worst case, your API requests will be submitted sequentially.

To ensure that all of your requests are submitted in parallel, and that all responses have been received, wrap your calls to map (both the request and the deref) inside a doall, which will force evaluation of the lazy sequence.

Upvotes: 5

Related Questions