I have a bunch of servers with multiple instances accessing a resource that has a hard limit on requests per second. I need a mechanism to lock the access on this resource for all servers and instances that are running. There is a restful distributed lock manager I found on github: https://github.com/thefab/restful-distributed-lock-manager Unfortunately there seems to be a min. lock time of 1 second and it's relatively unreliable. In several tests it took between 1 and 3 seconds to unlock a 1 second lock. Is there something well tested with a python interface I can use for this purpose? Edit: I need something that auto unlocks in under 1 second. The lock will never be released in my code.

Answers (7)

Reputation: 3805

Alternatively, AWS DynamoDB can be used to implement a locking protocol and a libre implementation exists for python

import boto3
from python_dynamodb_lock.python_dynamodb_lock import DynamoDBLockClient


dynamodb = boto3.resource("dynamodb")

lock_client = DynamoDBLockClient(dynamodb_resource)

with lock_client.acquire_lock("my-key"):
    pass  # ... app logic that requires the lock ...

The project is sort of stale and very tailored for an AWS environment, but I think it is worth being mentioned in this question.

See

Upvotes: 0

N1ngu

Reputation: 3805

Alternatively, a poor man solution is to just use filelock on top of a distributed file system! E.g. nfs, s3fs or whatnot.

import filelock

lock = filelock.FileLock("/path/to/mounted/dfs/my-id")
with lock:
    pass  # do your thing

Be cautious with your choice: the consistency of such lock will be in pair with the distributed file system's specific consistency.

Upvotes: 0

Keyless

Reputation: 106

The lowest effort way to implement this is to use lockable.

It offers low-level lock semantics and it comes with a Python client. Iportantly, you don't need to set up any database or server, it works by storing the lock on the lockable servers.

Locks have variable TTLs, but you can also release them early:

$ pip install lockable-dev

from lockable import Lock

my_lock = Lock('my-lock-name')

# acquire the lock
my_lock.acquire()

# release the lock
my_lock.release()

Upvotes: 2

David S

Reputation: 19

The distributed lock manager Taooka http://taooka.com has a TTL accuracy to nanoseconds. But it only has Golang client library.

Upvotes: -1

Alexander Gryanko

Reputation: 682

For my cluster I'm using ZooKeeper with python-kazoo library for queues and locks.

Modified example from kazoo api documentation for your purpose: http://kazoo.readthedocs.org/en/latest/api/recipe/lock.html

zk = KazooClient()
lock = zk.Lock("/lockpath", "my-identifier")
if lock.acquire(timeout=1):

   code here

   lock.release()

But you need at least three nodes for ZooKeeper as I remember.

Upvotes: 1

Jan Vlcinsky

Reputation: 44102

My first idea was using Redis. But there are more great tools and some are even lighter, so my solution builds on zmq. For this reason you do not have to run Redis, it is enough to run small Python script.

Requirements Review

Let me review your requirements before describing solution.

limit number of request to some resource to a number of requests within fixed period of time.
auto unlocking
resource (auto) unlocking shall happen in time shorter than 1 second.
it shall be distributed. I will assume, you mean that multiple distributed servers consuming some resource shall be able and it is fine to have just one locker service (more on it at Conclusions)

Concept

Limit number of requests within timeslot

Timeslot can be a second, more seconds, or shorter time. The only limitation is precision of time measurement in Python.

If your resource has hard limit defined per second, you shall use timeslot 1.0

Monitoring number of requests per timeslot until next one starts

With first request for accessing your resource, set up start time for next timeslot and initialize request counter.

With each request, increase request counter (for current time slot) and allow the request unless you have reached max number of allowed requests in current time slot.

Serve using zmq with REQ/REP

Your consuming servers could be spread across more computers. To provide access to LockerServer, you will use zmq.

Sample code

zmqlocker.py:

import time
import zmq

class Locker():
    def __init__(self, max_requests=1, in_seconds=1.0):
        self.max_requests = max_requests
        self.in_seconds = in_seconds
        self.requests = 0
        now = time.time()
        self.next_slot = now + in_seconds

    def __iter__(self):
        return self

    def next(self):
        now = time.time()
        if now > self.next_slot:
            self.requests = 0
            self.next_slot = now + self.in_seconds
        if self.requests < self.max_requests:
            self.requests += 1
            return "go"
        else:
            return "sorry"


class LockerServer():
    def __init__(self, max_requests=1, in_seconds=1.0, url="tcp://*:7777"):
        locker=Locker(max_requests, in_seconds)
        cnt = zmq.Context()
        sck = cnt.socket(zmq.REP)
        sck.bind(url)
        while True:
            msg = sck.recv()
            sck.send(locker.next())

class LockerClient():
    def __init__(self, url="tcp://localhost:7777"):
        cnt = zmq.Context()
        self.sck = cnt.socket(zmq.REQ)
        self.sck.connect(url)
    def next(self):
        self.sck.send("let me go")
        return self.sck.recv()

Run your server:

run_server.py:

from zmqlocker import LockerServer

svr = LockerServer(max_requests=5, in_seconds=0.8)

From command line:

$ python run_server.py

This will start serving locker service on default port 7777 on localhost.

Run your clients

run_client.py:

from zmqlocker import LockerClient
import time

locker_cli = LockerClient()

for i in xrange(100):
    print time.time(), locker_cli.next()
    time.sleep(0.1)

From command line:

$ python run_client.py

You shall see "go", "go", "sorry"... responses printed.

Try running more clients.

A bit of stress testing

You may start clients first and server later on. Clients will block until the server is up, and then will happily run.

Conclusions

described requirements are fulfilled
- number of requests is limited
- no need to unlock, it allows more requests as soon as there is next time slot available
- LockerService is available over network or local sockets.
it shall be reliable, zmq is mature solution, python code is rather simple
it does not require time synchronization across all participants
performance will be very good

On the other hand, you may find, that limits of your resource are not so predictable as you assume, so be prepared to play with parameters to find proper balance and be always prepared for exceptions from this side.

There is also some space for optimization of providing "locks" - e.g. if locker runs out of allowed requests, but current timeslot is already almost completed, you might consider waiting a bit with your "sorry" and after a fraction of second provide "go".

Extending it to real distributed lock manager

By "distributed" we might also understand multiple locker servers running together. This is more difficult to do, but is also possible. zmq allows very easy connection to multiple urls, so clients could really easily connect to multiple locker servers. There is a question, how to coordinate locker servers not to allow too many request to your resource. zmq allows inter-server communication. One model could be, that each locker server would publish each provided "go" on PUB/SUB. All other locker servers would be subscribed, and used each "go" to increase their local request counter (with a bit modified logic).

Upvotes: 25

h4unt3r

Reputation: 844

Your requirements seem very specific. I'd consider writing a simple lock server then implementing the locks client side with a class that acquires a lock when it is created then deletes the lock when it goes out of scope.

class Lock(object):
    def __init__(self,resource):
        print "Lock acquired for",resource
        # Connect to lock server and acquire resource

    def __del__(self):
        print "Lock released"
        # Connect to lock server and unlock resource if locked

def callWithLock(resource,call,*args,**kwargs):
    lock = Lock(resource)
    return call( *args, **kwargs )

def test( asdf, something="Else" ):
    return asdf + " " + something

if __name__ == "__main__":
    import sys
    print "Calling test:",callWithLock( "resource.test", test, sys.argv[0] )

Sample output

$ python locktest.py 
Calling test: Lock acquired for resource.test
Lock released
locktest.py Else