How to lock documents while using couchbase while using spring scheduler

Question

We have a microservice which has an endpoint which takes a request, creates a document in couchbase and returns OK. The document is later processed by a spring-scheduler.

We are using docker to deploy the microservice. Now, if we have more than one instance of the app running then we have a problem where each instance will read and process the same document which is not required. At a time only one instance should process the document.

1. We first thought of using an 'is-scheduled' flag and optimistic locking to avoid other instances from processing it however, if some instance which has read and set the flag crashes we will have documents with is-scheduled as true forever and they will never be picked up for processing.
2. We also thought of using pessimistic locking, however the couchbase sdk only allows locking for 30 secs. We cannot guarantee the processing to complete in 30secs.
3. We thought of creating a document with a similar id but a different prefix with a TTL. Before getting a document for processing we will try to get this document (which is a lock, sort of). If the document is available means we can skip the documents processing. Even if the application crashes this lock will expire and the document can be processed later. So far we haven't found any problems with this solution.

Questions :

1. For the second point, is their a way in couchbase where we can get the lock on a document for more than 30 secs say 4 hours or more.
2. Is their any corner case missed in the 3rd solution?
3. Or is their a better way to solve this problem?

Update : We went ahead with the 3rd solution. Its was very simple to implement and seems to satisfy all the cases that we know of.

Answer 1

You could pre-designate the documents to specific back-end processors. When each document is created, add a field to it stating which processor should be responsible for it.