CODEWITHSUNDEEP
algorithmmongoosebeanstalkd
user4170419
user4170419

Reputation:

Algorithm for beanstalkd queue

I am processing huge amount of parcels for tracking. I have mongodb in which each users have different parcels. There are more than 20 companies(tnt,fedex,dhl etc) and each company only allows single tracking or batch tracking with a limit (e.g tnt=1, fedex=20 and dhl=10)

I am trying to find a way to queue the tracking jobs which effectively uses the batch processing. For example if I have total 50 fedex jobs, 3 tnt jobs and 11 dhl jobs, I want to queue them like below

[fedex job:[20 jobs]][fedex job:[20 jobs]][fedex job:[10 jobs]]
[tnt job:[1 job]][tnt job:[1 job]][tnt job:[1 job]]
[dhl job:[10 job]][dhl job:[1 job]]

How can I achieve above job scenario?

EDIT

I want to able to run this on beanstalkd with a single or multiple workers. Therefore independent state is very important. Beanstalkd jobs should look like below,

{
'slug':'fedex'
'trackingNumbers':[11,22,33....] //20
}

{
'slug':'tnt'
'trackingNumbers':[11,22,33....] //10
}

{
'slug':'dhl'
'trackingNumbers':[11] //1
}

{
'slug':'tnt'
'trackingNumbers':[11,22,33....] //10
}

{
'slug':'fedex'
'trackingNumbers':[11,22,33....] //20
},

{
'slug':'dhl'
'trackingNumbers':[11] //1
}

I don't know really maybe there is a MongoDB function to aggregate parcels of each users to above chunks.

Here is my Mongoose Models

var UserSchema = mongoose.Schema({
    email: {type: String, index: {unique: true, sparse: true}},
    password: String,
    createdOn: {type: Date, default:Date.now },
    purchaseDate:Date,
    expiryDate:Date,
    purchaseID: {type: String, index: {unique: true, sparse: true}},
    isEnabled: {type: Boolean,default: true}
});

UserSchema.virtual('parcels',{
    ref: 'Parcel',
    localField:'_id',
    foreignField:'owner'
});

var ParcelSchema = mongoose.Schema({
    owner : { type:mongoose.Schema.Types.ObjectId, ref: 'User' },
    dna:String,
    name: String,
    count: {type: Number, default: 1},
    trackingNumber: String,
    slug:String,
    isDelivered: {
        type: Boolean, default: false
    },
    startDate: {type: Date, default:Date.now },

    updateDate: Date,
    finishDate: Date,
    expectedDate: Date,
    extra:String,
    details:[{
        _id:false,
        message: String,
        location: String,
        date: Date,
        status: Number
    }]
});

Upvotes: 0

Views: 225

Answers (1)

Alexander Azarov
Alexander Azarov

Reputation: 13221

Here it is a suggestion. You could do the following:

  • have a tube per delivery company, e.g. fedex, dhl, etc.
  • have one worker attached to only one queue (but you may have as many workers as you would like)
  • every worker should use Beanstalkd's stats-tube <tube> command before reserve. stats-tube returns the number of currently reserved jobs in this queue (field current-jobs-reserved in YAML output of the command)
  • so your worker makes sure it doesn't violate SLA for the queue it processes

Upvotes: 0

Related Questions