CODEWITHSUNDEEP
concurrencyclojureparallel-processingfuture
Matthew Gretton
Matthew Gretton

Reputation: 333

Body of clojure future executed twice in 2 CPU environment

I'm seeing some unexpected behaviour when kicking off a clojure future in an environment with multiple CPUs, I do not see this behaviour running on my single CPU (multiple core) dev pc.

Essentially the code looks something like:

(future
  (while true
    (let [work-items (get-work ...)]
      (doall
        (for [w work-items]
          (process-work w)))
      (Thread/sleep 10000))))

get-work returns work items that have not yet been processed.

process-work processes, and saves down the fact that a work item has been processed.

The assumption here is that all work items that were returned by the call to get-work will be persisted to the database, which will mean the next time get-work is called these work items will not be returned.So all coordination is via the DB.

When I run this on my single CPU (multiple core) dev pc, the body of the future gets executed once as I would have expected. By once I mean that all of the available work items (for the first iteration of the while loop) are persisted once.

But when running this in an environment that has 2 CPUs (deploying into our companies QA env) it appears that the body of the future is being executed twice. By twice I mean that I see the all of the available work items persisted into the db twice.

I can't see anything in the definition of future that indicates this is expected behaviour. Has anyone seen this before? It's a frustrating one to test as I can only see it when I deploy into our QA environment.

In both cases, I am running against the same database, with the same run parameters. So the underlying data should be identical. I'm running on only one machine in the QA environment at the moment, and it is not load balanced.

I'll try and come up with a minimal example that demonstrates the behaviour, but until then any advice would be greatly appreciated.

Thanks,

Matt.

Upvotes: 0

Views: 113

Answers (1)

amalloy
amalloy

Reputation: 91907

You are focusing on the wrong thing. There is nothing about the number of cores that changes how many times future evaluates its body. Instead, figure out what else is different about the two setups you're deploying to. Does QA have more work-items or something?

Furthermore, what makes you think you know how many times it is executing? Given that the body is wrapped in a while true, all the stuff inside it could run any number of times: once or twice would both be a bit of a surprise to me, compared to "forever, until an exception happens".

Upvotes: 2

Related Questions