CODEWITHSUNDEEP
clojureparallel-processing
Kamil Lelonek
Kamil Lelonek

Reputation: 14744

How to perform synchronous parallel functions in Clojure?

I have an application which has an initial flow that can be done in parallel:

  1. Fetch two JSON documents (I use clj-http for that)
  2. Parse these documents (extract only required data)
  3. Join the results
  4. Dump them into a one file

So there's something like that:

some-entry-point
      /\
     /  \
    /    \
   /      \
fetchA   fetchB
  |         |
  |         |
parseA   parseB
  \        /
   \      /
    \    /
     \  /
      \/
     join
      |
      |
     dump

What is the correct and the most current way to achieve that?

What I found so far are:

Upvotes: 3

Views: 1006

Answers (1)

Leonid Beschastny
Leonid Beschastny

Reputation: 51450

Since you have exactly two branches here, it'll be best to dispatch parallel jobs to separate threads using future function. future will return you a future object (a special promise which will be automatically resolved when the job will be completed).

Here is how it will look:

(defn some-entry-point
  [obja objb]
  (let [pa (-> (fetchA obja)
               parseA
               future)
        pb (-> (fetchB objb)
               parseB
               future)]
    (-> (join @pa @pb)
        dump)))

@ here is a shortcut (reader macro) for deref function.

Or course, you could execute one of you branches in main thread, creating only one future object:

(defn some-entry-point
  [obja objb]
  (let [p (-> (fetchB objb)
              parseB
              future)]
    (-> (fetchA obja)
        parseA
        (join @p)
        dump)))

You could also generalize this approach to fetch and parse multiple objects using some general fetch and parse functions, using pmap for paralelization:

(defn some-entry-point
  "Fetch and parse multiple objects in parallel"
  [objs]
  (->>  objs
        (pmap (comp parse fetch))
        (apply join)
        dump))

Upvotes: 4

Related Questions