CODEWITHSUNDEEP
multithreadingclojurelocking
Dr Ljotsson
Dr Ljotsson

Reputation: 492

Achieving multiple locks in clojure

I'm new to Clojure and am writing a web application. It includes a function fn performed on user user-id which includes several steps of reading and writing to the database and file system. These steps cannot be performed simultaneously by multiple threads (will cause database and file system inconsistencies) and I don't believe they can be performed using a database transaction. However, they are specific to one user and thus can be performed simultaneously for different users.

Thus, if a http request is made to perform fn for a specific user-id I need to make sure that it is completed before any http requests can perform fn for this user-id

I've come up with a solution that seems to work in the REPL but have not tried it in the web server yet. However, being unexperienced with Clojure and threaded programming I'm not sure whether this is a good or safe way to solve the problem. The following code has been developed by trial-and-error and uses the locking function - which seems to go against the "no locks" philosophy of Clojure.

(ns locking.core)    

;;; Check if var representing lock exists in namespace
;;; If not, create it. Creating a new var if one already
;;; exists seems to break the locking.
(defn create-lock-var
  [var-name value]
  (let [var-sym (symbol var-name)]
    (do
      (when (nil? (ns-resolve 'locking.core var-sym))
        (intern 'locking.core var-sym value))
      ;; Return lock var
      (ns-resolve 'locking.core var-sym))))

;;; Takes an id which represents the lock and the function
;;; which may only run in one thread at a time for a specific id
(defn lock-function
  [lock-id transaction]
  (let [lock (create-lock-var (str "lock-id-" lock-id) lock-id)]
    (future
      (locking lock
        (transaction)))))

;;; A function to test the locking
(defn test-transaction
  [transaction-count sleep]
  (dotimes [x transaction-count]
    (Thread/sleep sleep)
    (println "performing operation" x)))

If I open three windows in REPL and execute these functions, it works

repl1 > (lock-function 1 #(test-transaction 10 1000)) ; executes immediately
repl2 > (lock-function 1 #(test-transaction 10 1000)) ; waits for repl1 to finish
repl2 > (lock-function 2 #(test-transaction 10 1000)) ; executes immediately because id=2

Is this reliable? Are there better ways to solve the problem?

UPDATE

As pointed out, the creation of the lock variable is not atomic. I've re-written the lock-function function and it seems to work (no need for create-lock-var)

(def locks (atom {}))

(defn lock-transaction
  [lock-id transaction]
  (let [lock-key (keyword (str "lock-id-" lock-id))]
    (do
      (compare-and-set! locks (dissoc @locks lock-key) (assoc @locks lock-key lock-id))
      (future
        (locking (lock-key @locks)
          (transaction))))))

Note: Renamed the function to lock-transaction, seems more appropriate.

Upvotes: 0

Views: 327

Answers (2)

Valentin Waeselynck
Valentin Waeselynck

Reputation: 6061

Since you're making a web app, I have to warn you: even if you do manage to get in-process locking right (which is not easy in itself), it will be for nothing as soon as you deploy your web server on more than one machine (which is almost mandatory if you want your app to be highly-available).

So basically, if you want to use locking, you'd better use distributed locking. From this point on, this discussion is not Clojure-specific, since Clojure's concurrency tools won't be especially helpful here.

For distributed locking, you could use something like Zookeeper. If you don't want to set up a whole Zookeeper cluster just for this, maybe you can compromise by using a Redis database (the Carmine library gives you distributed locks out of the box), although last time I heard Redis locking is not 100% reliable.

Now, it seems to me locking is not especially a requirement, and is not the best approach, especially if you're striving for idiomatic Clojure. How about using a queue instead ? Some popular JVM message brokers (such as HornetQ and ActiveMQ) give you Message Grouping, which guarantees that messages of the same group-id will be processed (serially) by the same consumer. All you have to do is have some threads listen to the right queue and set the user-id as the group id for your messages.

HACK: If you don't want to set up a distributed message broker, maybe you could get around by enabling sticky sessions on you load balancer, and using such a message broker in-VM.

By the way, don't name your function fn :).

Upvotes: 1

amalloy
amalloy

Reputation: 92117

Don't use N vars in a namespace, use an atom wrapped around 1 hash-map mapping N symbols to N locks. This fixes your current race condition, avoids creating a bunch of silly vars, and is easier to write anyway.

Upvotes: 5

Related Questions